Home » How to determine the maximum number to pass to make -j option?

How to determine the maximum number to pass to make -j option?

Solutons:


nproc gives the number of CPU cores/threads available, e.g. 8 on a quad-core CPU supporting two-way SMT.

The number of jobs you can run in parallel with make using the -j option depends on a number of factors:

  • the amount of available memory
  • the amount of memory used by each make job
  • the extent to which make jobs are I/O- or CPU-bound

make -j$(nproc) is a decent place to start, but you can usually use higher values, as long as you don’t exhaust your available memory and start thrashing.

For really fast builds, if you have enough memory, I recommend using a tmpfs, that way most jobs will be CPU-bound and make -j$(nproc) will work as fast as possible.

The most straight-foward way is to use nproc like so:

make -j`nproc`

The command nproc will return the number of cores on your machine. By wrapping it in the ticks, the nproc command will execute first, return a number and that number will be passed into make.

You may have some anecdotal experience where doing core-count + 1 results in faster compile times. This has more to do with factors like I/O delays, other resource delays and other availability of resource constraints.

To do this with nproc+1, try this:

make -j$((`nproc`+1))

Unfortunately even different portions of the same build may be optimal with conflicting j factor values, depending on what’s being built, how, which of the system resources are the bottleneck at that time, what else is happening on the build machine, what’s going on in the network (if using distributed build techniques), status/location/performance of the many caching systems involved in a build, etc.

Compiling 100 tiny C files may be faster than compiling a single huge one, or viceversa. Building small highly convoluted code can be slower than building huge amounts of straight-forward/linear code.

Even the context of the build matters – using a j factor optimized for builds on dedicated servers fine tuned for exclusive, non-overlapping builds may yield very dissapointing results when used by developers building in parallel on the same shared server (each such build may take more time than all of them combined if serialized) or on servers with different hardware configurations or virtualized.

There’s also the aspect of correctness of the build specification. Very complex builds may have race conditions causing intermittent build failures with occurence rates that can vary wildly with the increase or decrease of the j factor.

I can go on and on. The point is that you have to actually evaluate your build in your very context for which you want the j factor optimized. @Jeff Schaller’s comment applies: iterate until you find your best fit. Personally I’d start from the nproc value, try upwards first and downwards only if the upwards attempts show immediate degradation.

Might be a good idea to first measure several identical builds in supposedly identical contexts just to get an idea of the variability of your measurements – if too high it could jeopardise your entire optimisation effort (a 20% variability would completely eclipse a 10% improvement/degradation reading in the j factor search).

Lastly, IMHO it’s better to use an (adaptive) jobserver if supported and available instead of a fixed j factor – it consistently provides a better build performance across wider ranges of contexts.

Related Solutions

StringBuilder delete methods [closed]

I'd say this is fine. The intent is different. Lets try this for example: String builder sb = new StringBuilder(); sb.append("hello").append("hello"); sb.delete(0,5); sb.toString(); It prints "hello". Had you read the documentation, you'd see that the end is...

How to get specific element from array of type any in swift?

You should not be using JSONSerializer in Swift. A better option is a JSONDecoder. JSONDecoder will allow you to establish and preserve the type information for the data you've decoded. Here is an Playground example of how you would use JSONDecoder to handle...

How to make RecyclerView behave like Paragraph?

The answer is FlexBoxLayoutManager you can use it with RecyclerView put implementation 'com.google.android:flexbox:1.0.0' in your gradle file. Now setting up RecyclerView layout, <?xml version="1.0" encoding="utf-8"?> <RelativeLayout...

Piping commands after a piped xargs

You are almost there. In your last command, you can use -I to do the ls correctly -I replace-str Replace occurrences of replace-str in the initial-arguments with names read from standard input.  Also, unquoted blanks do not terminate input items; instead the...

create arrow using css

Try to use Font awsome LINK HTML: <i class="fa fa-arrow-right"></i> fa-arrow-right Yes, there are several ways of doing this. 1. With an image First of all, in order to do what you've suggested, you can indeed use an image - but instead of coloring...

accessing values structure within structure? [closed]

Just create an object of emp1 inside your main, and give it a value like below and it should be ok emp1 emp; emp.number.number1=3; //printf then with emp.number.number1 😉 You are accessing the value "number1" like emp1.number.number1 but here "emp" data type...

What’s wrong with this way to add rows to dataframe?

Try: data <- data.frame(A= character(0), B= character(0), stringsAsFactors=FALSE) data[1, ] <- c("AA", "BB") In your example, A and B are factors. I would suggest a different approach from the other answers. If A and B are something like names or ID...

Panagram python code [closed]

import string def isPanagram(sentence): alphabet = string.lowercase[:26] # list of letters a-z return all(letter in sentence.lower() for letter in alphabet) The function all will return true if all elements in a list are true The list comprehension inside goes...

VB Code explanation? [closed]

Declare a student name variable that we will call studentname as a string. Think of variables as you would do x and y in an algebraic equation. They are simply names for things of a particular type. In a maths equation x and y are numbers. A string is a...

remove null items from a multidimensional array in C#

Are your null objects always at the end of the multidimentional array? If so use something like this to remove the null items: private static void ReDimMultiObjects(ref object[,] arr, int length) { object[,] arrTemp = new object[length, 3];//New number of...

Check if string contains only one type of letter C# [closed]

Try this: private bool ContainsOnlyOneLetter(string String) { if(String.Length == 0) { return true; } for(int i = 0; i < String.Length;i++) { if(String.Substring(i,1) != String.Substring(0,1)) { return false; } } return true; } And you can use the function...

Couldn’t understand this SWIFT Fucntion [closed]

You're getting 120 because you're asking the tuple for the sum value twice. The tuple passed back is (min: Int, max: Int, sum: Int). In this case, sum is also index number 2 (min is index zero, max is index one). So in this case statistics.sum == statistics.2...

C++ Cli code stucked [closed]

The code you provided does not work on any numbers. The only thing it does is construct some SQL query check its result During the SQL-query-construction, the code does not assume any "numbers" to be provided. It actually uses a variable called...