As avid researchers delving into the depths of Kubernetes infrastructure, we embarked on a journey to review the security posture of ChartMuseum, the beloved Helm chart repository server. Little did we know this exploration would unveil an interesting revelation of Kubernetes security practices.

During our investigation, a particular observation seized our attention – an exposed API endpoint within ChartMuseum. What initially seemed innocuous soon revealed itself to be a potential Achilles’ heel in the fortifications of Kubernetes deployments. At this exposed endpoint, lay a security concern that allowed any unauthenticated user to freely access, download, and scour through Helm charts, including those harboring sensitive configuration settings.

As we delved deeper, we realized the gravity of this discovery. Such unrestricted access could expose critical infrastructure secrets, compromising data integrity, and undermining the very foundations of secure Kubernetes environments and the associated systems and applications.

In this blog series, we aim to shed light on this discovery within ChartMuseum’s API, elucidating its implications, and advocating for additional security measures to mitigate such risks. Together, we’ll explore the concern and how the risk can be mitigated for your own deployments.

We initially came across this service during our attack surface monitoring, and it seemed benign at first with the “Welcome to ChartMuseum!” landing page. Upon further exploration and researching the ChartMuseum documentation, we discovered a number of API endpoints available to interact with the service.

Some of the available endpoints are:

Server Info

• GET / - HTML welcome page
• GET /info - returns current ChartMuseum version
• GET /health - returns 200 OK

Helm Chart Repository

• GET /index.yaml - retrieved when you run helm repo add chartmuseum
• GET /charts/mychart-0.1.0.tgz - retrieved when you run helm install chartmuseum/mychart
• GET /charts/mychart-0.1.0.tgz.prov - retrieved when you run helm install with the –verify flag

Chart Manipulation

• POST /api/charts - upload a new chart version
• POST /api/prov - upload a new provenance file
• DELETE /api/charts/<name>/<version> - delete a chart version (and corresponding provenance file)
• GET /api/charts - list all charts
• GET /api/charts/<name> - list all versions of a chart
• GET /api/charts/<name>/<version> - describe a chart version
• HEAD /api/charts/<name> - check if chart exists (any versions)
• HEAD /api/charts/<name>/<version> - check if chart version exists

The main endpoints that draw our attention are the Chart Manipulation endpoints, in particular, the GET /api/charts endpoint.

By default, these endpoints do not enforce authentication. So, what is the importance of this endpoint? For starters, these endpoints show a listing of all the available charts to download. You may be wondering, what exactly is a chart in the context of ChartMuseum and Helm? A chart refers to a packaged collection of Kubernetes resources that defines a set of related applications, services, and configurations. Think of it as a bundle or package that encapsulates everything needed to deploy and manage an application on a Kubernetes cluster.

The following is a breakdown of what typically is the genetic makeup of a chart:

• Manifests: Helm charts contain Kubernetes YAML manifests that define the desired state of resources within a Kubernetes cluster. These manifests can include deployments, services, ingress rules, config maps, secrets, and any other Kubernetes objects required to run the application.

• Templates: Helm charts often include templated YAML files. Helm uses Go’s template language to allow for dynamic generation of Kubernetes manifests. This enables parameterization and customization of the manifests based on configuration values provided during chart installation.

• Values File: Charts usually come with a values.yaml file, which contains default configuration values for the application. Users can override these values during installation to customize the deployment according to their specific requirements.

• Metadata: Each Helm chart includes metadata such as the chart name, version, description, maintainers, and dependencies. This metadata helps users understand the purpose of the chart and its compatibility with different environments.

• Dependencies: Helm charts can specify dependencies on other charts. This allows complex applications to be composed of multiple smaller charts, simplifying management and deployment. Helm automatically resolves dependencies and installs them along with the main chart.

Now that this is out of the way, back to the importance of the chart API. The data is formatted in JSON with the following data sets:

• Name
• Home
• Version
• Description
• Keywords
• Maintainers
• Icon
• API Version
• App Version
• Type
• URLs
• Created
• Digest

The main data set that we are interested in is URLs. This will give us the full path to download each package.

Once you download the package, just open the tarfile and start hunting for juicy configuration information such as usernames, passwords, and API keys.

This is simple enough to download if you have 1 or 2 packages. But what if you have hundreds or thousands to comb through?

We are firm believers in first doing tasks manually. We understood the problem with manually downloading hundreds or thousands of packages and searching for sensitive information. So, we built a small little program called ChartSearch that will accept a particular URL and a keyword(s) that will auto download all the packages, perform a text search in each package and display the results and the location of the keyword hit in its respective directory. The script will default to searching for key-value pairs as the output is cleaner. However, there is a search for free text using the –free-text switch. Feel free to download and use ChartSearch here.

We wanted to see how prevalent ChartMuseum exposures were on the internet, so we leveraged Shodan searches to assess the usage. Surprisingly, it was not that bad. 225 instances according to Shodan. We suspect as Kubernetes deployments expand due to the AI demand; many more instances will be exposed unless awareness about this issue is shared. But we also need to acknowledge that ChartMuseum instances are probably deployed internally with similar setups.

What Actions Must We Take?

First, understand the business justification for having ChartMuseum instances open to the internet. If internet access is a business requirement, ensure sensitive settings are not hardcoded in the configurations. Ideally through an automated process as we’re all prone to make mistakes.

Second, consider enabling basic-authentication to protect all API routes for the internet accessible ChartMusuem instances. ChartMuseum supports both basic authentication and Bearer/Token Authentication. Please reference https://chartmuseum.com/docs/#basic-auth and https://chartmuseum.com/docs/#bearer-token-auth.

Third, if you do decide to configure authentication, make sure to only allow HTTPS since basic-auth credentials can be captured in the clear over HTTP. Please reference https://chartmuseum.com/docs/#https