CyberArk Conjur follower, system error and reboot during configuration: how troubleshoot it and verify the Postgres communication in the right way

During this period working on a CyberArk Conjur environment, we experienced a strange behavior during Conjur follower setup. 

The setup on the follower was starting, the seed received, imported, and expanded but after some other steps the process was hanging and ending with a generic "System Error".

After the error message, the Conjur follower was restarted. 

We did some troubleshooting inside the Conjur Follower pod and we have verified that the Follower was able to connect to the Conjur API leader successfully but it wasn't able to connect to the Postgres database and finish the initial replication.

The correct way to verify the Postgres connectivity from the follower to the leader is the following command:

echo "" | openssl s_client -starttls postgres -connect <lb_DNS>:5432 -showcerts

if the server certificate will be returned, the Postgres connectivity will work as expected.

In our case, we were unable to get the certificate, so this points us to the network LB where a colleague fixed the issue.

Thanks to CyberArk support which provided us the openssl command, easy to be executed from the container or any server. 

We did the same verification in another way but the openssl s_client could be found easily on containers and servers.

To have more explanation about openssl s_client and see more options, check this nice blog post.

CyberArk Vault Synchronizer - CASVM035E Vault name is missing , how fix it

As you know one of the piace of Cybeark Conjur architecture is Synchronizer which is needed to receive the secrets from the VAUL.

Last week, I took charge of an abandoned Synchronizer 11.7 that was not working for a while and needed also to upgrade to the newest 12.7.

After the upgrade (check this link for the steps ), the windows service was unable to start and the log contained the following error:

[5] [main] FATAL VaultConjurSynchronizer.Service.SynchronizerService - VCSS006F Failed to start CyberArk Vault-Conjur Synchronizer Service: CASVM035E Vault name is missing.

After a search on google, we were pointed to the CASOS log error for the VAULT where documentations suggest calling CyberArk support .

Luckily in the case of a  Synchronizer, we fixed it easily by editing the file

C:\Program Files\CyberArk\Synchronizer\VaultConjurSynchronizer.exe.config

where the value of the key  INTEGRATION_VAULT_NAME. was blank, after we have filled it with the vault name (present in vault.ini) the service was able to run again correctly and the secrets sync was made.

CyberArk Impact 2022 world tour ! I'll be there and you ?

As you probably know CyberArk has a big annual event named Impact, and this year was run in Boston.

Over the past weeks, CyberArk has announced a great initiative named CyberArk Impact world tour that will be hosted in several cities around the globe.

If you are interested, here on this page you could find all the details:

• cities involved
• agenda
• registration form

Personally, I'll attend the one in Milan scheduled for the 11th of October 2022.

I'm excited to have this opportunity to follow some interesting sessions and meet interesting people in person, see you there!

CyberArk Conjur , authenticators and integrations

During the past few weeks, I have described what a secrets manager is and I have given an overview of the architecture and system requirements of CyberArk Conjur. 

A secrets manager can't do his job if he can't talk to those who need to ask him for secrets, and that's where Conjur's magic comes in!

The "authenticators" are responsible of the authentication process in Conjur and they are specialised to make it in the most secure way depending on the service. 

Here is the list of the authenticators currently supported :

authn	Defines the Conjur default authenticator. Authentication for both users and hosts is based on an ID and API key.
authn-oidc	Leverages the identity layer provided by OIDC to allow applications to authenticate with Conjur and retrieve secrets needed for connecting to services such as a database.
authn-iam	Enables an AWS resource to use its AWS IAM role to authenticate with Conjur.
authn-azure	Enables an Azure resource to authenticate with Conjur
authn-jwt	Enables an application to authenticate to Conjur using a JWT from a JWT Provider.
authn-gcp	Enables a Google Cloud Platform resource to authenticate with Conjur
authn-k8s	Authenticates hosts that are Kubernetes resources, such as a Kubernetes namespace, deployment, stateful set, and others. Authentication is certificate-based using a mutual TLS connection.
authn-ldap	Authenticates users based on an LDAP directory.

By default, after the initial setup, authn is the only authenticator enabled and it's responsible of the Conjur access with username or API key (randomly generated between 51 and 56 characters). 

When integrations are needed, for example with a Kubernetes cluster, you need to activate and configurate the authn-k8s authenticator. 

This authenticator can establish a secure mTLS connection compliant with Spiffie framework, as described in the following diagram (click here for more details): 

If  you need integrations with cloud identities supporting JWT authentications, like Google Apigee, you will enable the authn-jwt authenticator that will perform the integrations as below:

These were just two examples within the multiple possible secure integration allowed by the authenticators, the following is more complete list:

• Kubernetes (locali o cloud) 
• API gateway
• Jenkins
• Ansible
• Puppet
• Terraform
• Cloud services

If the scenario described sound interesting for your needs, Conjur can definitely be the right choice, so contact your sales representative at CyberArk or your preferred CyberArk business partner..

CyberArk Conjur : a quick overview about architecture and system requirements

As I wrote in my last post, CyberArk Conjur is an enterprise secrets manager.  

In this post I'll do an architecture overview with some of the system requirements.

Conjur is currently available in 2 different versions , Enterprise and open source (known as OSS); the "cloud" version will be available soon, offered as a SAAS solution.

My post is about the enterprise version, that is similar but not the same as OSS.

Architecture

Conjur is distributed as a docker image, so deployment and configuration are fast and safe to be done.

This are the main services included in the appliance image: 

• nginx   - allows the access to the administrative GUI and the access to the API set to interact with the secrets
• postgres database - 2 databases, one for configurations and secrets, one for the audit logs
• conjur appliance - the product
• syslog-ng - it's used to collect audit and access logs

The enterprise smallest common configuration of Conjur is a 3 node cluster with auto-failover, as in the following diagram:

The cluster architecture is an active-passive cluster and the high availability is guaranteed by a load balancer put in front of the cluster, that switches the node accessed by the applications in case of failure.

Inside the cluster, a node will act as the "leader node"; it takes charge of the read/write operations of the system and it is accessed by the applications using the API to interact with the secrets.

The secondary nodes are named "standbys" and could be sync or async nodes,  depending on the postgres replication setup.  These nodes are inactive and do not provide the API service.

In case the leader node fails, after a time set into the configuration, the standbys will elect a new leader using etcd and the Rafts consensus algorithm. 

In case the auto-failover were not configured, a manual promotion is needed.

It's also possible to set a primary site configured with the auto-failover policy and a secondary site with async standbys, without the auto-failover, ready to be promoted manually. 

The last piece of the architecture is the "follower", a read only replica of the Leader. 

Followers, which allow secrets reads at scale, are horizontally-scaling components that are typically configured behind a load balancer to handle all types of read requests, including authentication, permission checks, and secret fetches.

Usually Followers are installed on a Kubernetes cluster to serve applications with a low latency response time.

The architecture diagram with followers could be expanded as below:

System requirements

The resources suggested for conjur nodes  for a production environments are: 

• 4 core
• 8 GB ram
• 50 GB hd

For a test environment:

• 2 core
• 4 GB ram
• 20 GB hd

This are the supported container runtime 

• docker 1.13 or later
• Mirantis Container runtime (MRC) 19.x, 20.10 su RHEL 8.x
• Kubernetes e Openshift (only for the follwer) 
• podman 3.3 e 3.4 

the host server should run an os supported by the container runtime (the most popular are RHEL  server, RHEL based or Ubuntu server LTS) .

In case the customer environment contains also a Cyberark Vault server that needs to be synched with Conjur, the following are the requirements for a CyberArk Vault Synchronizer server:   

• 4 core
• 8 GB ram
• OS Windows 2012 R2 or later

CyberArk Conjur - why you (probably) need an enterprise secrets manager

Security is always a complex topic to address because an error or omission in the processes can lead to serious economic or brand reputation damage for a company.

As secrets we could consider the following examples:

• usernames
• database passwords
• SSL certificates and keys
• SSH Keys
• cloud credentials 

reading the list of what could be considered a secret could easily explain why this topic needs to be considered and handled in the correct way.

Some of the bad practices or risk related could include:

• hardcoded secrets in the code
• data-breach
• password leak
• secrets pushed in public repository

with practice like lateral movement, only one secret compromised could be enough to compromise an environment. 

To help and prevent bad situations and risks there are tools named "enterprise secrets manager" and now I'd like to start a series of posts about  CyberArk Conjur on this blog.

Conjur permits getting rid of the direct use of the secrets and using a set of API rest is a programmable tool and could be accessed using URL or open source tools. 

The security is granted through security policies without slowing the speed of the developers involved.

The whole corporate security could be improved with the use of rotators that are able to programmatically change secrets value. 

In case other Cyberark software like Pas Vault Conjur is already implemented, Conjur could be integrated using the Synchroniser component giving the usual level of security to the Cloud native infrastructure.

Conjur it's available in 2 different versions, enterprise and opensource with some different functionality.

During the next posts, I will explain details about architecture, secrets management, and product news . 

I've stared a new journey as Senior DevSecOps Engineer

Hello there , how are you ? 

I'm really good !!! As you may have read on my social , starting from 16th May I've started a new position as Senior DevSecOps for Sighup. 

I'm really exited about this new working opportunity and I'm writing this post because this will also have effect on this blog topic that will change from the previous topics to cloud native infrastructure security , starting with tools like CyberArk Conjur .

The previous content of this blog will stay here forever, because I think that could be helpful for a while  and I've the intention to honor my HCL Ambassador role. 

I love to add that my current role of my LetsConnect user group organizer will be unchanged so I hope to meet you all again during our  next event.