You can adjust a banjo nut with a few simple hand tools and a lot of patience. Begin by taking accurate measurements of the fret height on your banjo neck. Once measured, arrange a set of feeler gauges to this height and then add another very thin feeler gauge to the top of the stack. Using a small file, the string grooves in the banjo nut can be deepened to allow the strings to ride closer to the banjo neck. You must determine how close you want your strings to be on the nut, in relation to the frets, without buzzing.
The typical adjustment of a banjo nut is high off of the frets, as the instrument is shipped this way from the factory. This makes the instrument difficult to play, and can leave your fingers sore after a short time of playing. The high adjustment is commonly done to avoid fret buzz associated with a nut that is too close to the fret's height. Many music shops offer professional setup on new purchases; however, you can do this yourself with only minor effort and a little patience. The key is to work slowly while taking frequent measurements and trial tunings for each string position that is worked on.
The key is to take accurate measurements of the fret height; this is the height from the fingerboard to the top of the wire frets. Once measured, a set of feeler gauges stacked to the proper height is placed on the fingerboard of the banjo neck directly behind the nut. You will need to decide on the additional height that you wish to have your strings set over the top of the frets, and place that size feeler gauge on top of the stacked feeler gauges.
Start by removing one string from the groove in the banjo nut and begin running a small file through the groove at a right angle to the tilted head stock of the banjo. Once the file makes contact with the feeler gauges, stop filing and slip the string back into the groove. Tighten and tune the string while listening for any signs of fret buzz. If you are satisfied with the new action of your banjo strings, continue on filing the remaining grooves in your banjo nut to the same depth. It is important to remember to repeat this process on the fifth string banjo nut as well.