YARN Containers Memory

Something worth checking before running any task or job, particularly with small clusters with not-so-powerful machines, is the Container Memory size in YARN. A Container in YARN represents a resource (memory and vcores) on a single node at a given cluster. Containers perform tasks.

In the Cloudera Manager, click on Clusters on the top menu and select YARN.

In the YARN menu, click on Configuration. In the search box just below, type memory container.

We should now see a list of cluster configuration items, including, Container Memory, Container Memory Minimum, Container Memory Maximum, etc.

Make sure that the value for the Container Memory field is at least 4 GB (this might be 1 GB by default if your EC2 instances have less than 8 GB memory). You can also set the Container Memory Maximum at 8 GB. Choose those values depending on the memory of your EC2 instances.

More details on Containers here: https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_ig_yarn_tuning.html