Home » How to troubleshoot DNS with systemd-resolved?

How to troubleshoot DNS with systemd-resolved?

Solutons:


Very helpful for troubleshooting is also:

journalctl -u systemd-resolved -f

There you can see what systemd-resolved is really doing.
In my case it was not contacting the DNS servers that were reported via systemd-resolve --status at all.
If it’s doing weird things like that, then somtimes a restart via sudo systemctl restart systemd-resolved is a good idea.

EDIT:

In order to get more information from resolved you need to put

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

into the override.conf of systemd-resolved via

sudo systemctl edit systemd-resolved

Restart to take effect:

sudo systemctl restart systemd-resolved

Use resolvectl status (systemd-resolve --status when using systemd version earlier than 239) to show your global and per-link DNS settings.

The resolv.conf file

The manual says:

The resolver is a set of routines in the C library that provide access to the Internet Domain Name System (DNS). The resolver config file contains information that is read by the resolver routines the first time they are invoked by a process.

If this file does not exist, only the name server on the local machine will be queried, and the search list contains the local domain name determined from the hostname.

systemd-resolved

On Ubuntu 20.04, systemd-resolved is a local DNS server included with systemd that acts as a stub resolver and it should automatically edit /etc/resolv.conf with the correct config.

On older versions you might need to manually symlink it with the following:

$ sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

By default, systemd-resolved configures a local DNS server listening at 127.0.0.53 on port 53 and you can run lsof -i @127.0.0.53:53 to verify. Make sure you do not have any other service like dnsmasq listening to the same address and port.

Note that 127.0.0.53 is the loopback interface but you can have another DNS server listening on port 53 at another IP address on the loopback interface, like 127.0.0.1 or 127.0.0.2.

If it does not work, try the following:

$ sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

This file contains DNS servers from DHCP leases but be aware that you will not benefit from the stub resolver caching feature.

dnsmasq with NetworkManager

The dnsmasq lightweight caching DNS server accepts DNS queries and either answers them from a small, local, cache or forwards them to a real, recursive, DNS server.

This is very good to have a little DNS service to test local websites for a web development machine.

First, verify that NetworkManager manages your current interface (wlp0s20f3 in my case):

$ nmcli dev
$ nmcli dev set wlp0s20f3 managed yes

Now you can enable the NetworkManager built-in dnsmasq resolver:

$ cat /etc/NetworkManager/NetworkManager.conf
[main]
dns=dnsmasq

You can now benefit from the flexibility of NetworkManager along dnsmasq, whose config is located in /etc/NetworkManager/dnsmasq.d:

$ cat /etc/NetworkManager/dnsmasq.d/custom.conf
# Resolve all domains ending in .dev to 127.0.0.1
address=/.dev/127.0.0.1

Do not forget to disable the system-wide dnsmasq service if present and apply your changes:

$ sudo systemctl stop dnsmasq.service && sudo systemctl disable dnsmasq.service
$ sudo systemctl restart NetworkManager.service

If for some reasons you prefer to have dnsmasq running separately, you must first prevent NetworkManager from overriding /etc/resolv.conf:

$ cat /etc/NetworkManager/NetworkManager.conf
[main]
dns=none

Now you must tell dnsmasq to get its upstream servers from somewhere other that /etc/resolv.conf:

$ cat /etc/dnsmasq.conf
listen-address=127.0.0.53
resolv-file=/run/NetworkManager/resolv.conf

This file is generated by NetworkManager and also contains DNS servers from DHCP leases. Now configure dnsmasq for resolving:

$ cat /etc/resolv.conf
# Use local dnsmasq resolver
nameserver 127.0.0.53

Configure your own local DNS settings:

$ cat /etc/dnsmasq.d/custom.conf
# Resolve all domains ending in .dev to 127.0.0.1
address=/.dev/127.0.0.1

Apply your new config:

$ sudo systemctl stop systemd-resolved.service && sudo systemctl disable systemd-resolved.service
$ sudo systemctl restart dnsmasq.service

dnsmasq with systemd-networkd

Unfortunately it is not as easy as with NetworkManager and this is due to the fact that systemd-networkd does not expose a file with DNS nameservers obtained from DHCP servers as this is managed by systemd-resolved instead.

This is also the case if you use the quite deprecated ifupdown package.

Open the man for dhclient-script:

$ man 8 dhclient-script

The HOOKS section says:

When it starts, the client script first defines a shell function, make_resolv_conf, which is later used to create the /etc/resolv.conf file. To override the default behaviour, redefine this function in the enter hook script.

So if you still want to use dnsmasq along systemd-networkd you will need to redefine make_resolv_conf to create a resolv-file for dnsmasq so that it can get its upstream servers from your DHCP settings.

The resolvconf package provides a wrapper interface around the different daemons involved in populating /etc/resolv.conf and you can use it to configure the make_resolv_conf behavior.


The same problem also happens if you need to obtain the IPv6 upstream servers from RDNSS in Router Advertisement messages (see Neighbor Discovery Protocol) as this is also managed by systemd-resolved.

Try the following command from the ndisc6 package:

$ rdisc6 wlp0s20f3

You should see various information coming from your router, including your IPv6 address prefix and DNS servers. If you do not see anything, your router may not support it.

The rdisc6 program implements the ICMPv6 Router Discovery in userland, using NETLINK_ROUTE sockets for which RDNSS support was added in Linux kernel 2.6.24. If you want to support older versions, you have no choices but to interact with the device driver at OSI Layer 2, using SOCK_RAW sockets.

Related Solutions

Joining bash arguments into single string with spaces

[*] I believe that this does what you want. It will put all the arguments in one string, separated by spaces, with single quotes around all: str="'$*'" $* produces all the scripts arguments separated by the first character of $IFS which, by default, is a space....

AddTransient, AddScoped and AddSingleton Services Differences

TL;DR Transient objects are always different; a new instance is provided to every controller and every service. Scoped objects are the same within a request, but different across different requests. Singleton objects are the same for every object and every...

How to download package not install it with apt-get command?

Use --download-only: sudo apt-get install --download-only pppoe This will download pppoe and any dependencies you need, and place them in /var/cache/apt/archives. That way a subsequent apt-get install pppoe will be able to complete without any extra downloads....

What defines the maximum size for a command single argument?

Answers Definitely not a bug. The parameter which defines the maximum size for one argument is MAX_ARG_STRLEN. There is no documentation for this parameter other than the comments in binfmts.h: /* * These are the maximum length and maximum number of strings...

Bulk rename, change prefix

I'd say the simplest it to just use the rename command which is common on many Linux distributions. There are two common versions of this command so check its man page to find which one you have: ## rename from Perl (common in Debian systems -- Ubuntu, Mint,...

Output from ls has newlines but displays on a single line. Why?

When you pipe the output, ls acts differently. This fact is hidden away in the info documentation: If standard output is a terminal, the output is in columns (sorted vertically) and control characters are output as question marks; otherwise, the output is...

mv: Move file only if destination does not exist

mv -vn file1 file2. This command will do what you want. You can skip -v if you want. -v makes it verbose - mv will tell you that it moved file if it moves it(useful, since there is possibility that file will not be moved) -n moves only if file2 does not exist....

Is it possible to store and query JSON in SQLite?

SQLite 3.9 introduced a new extension (JSON1) that allows you to easily work with JSON data . Also, it introduced support for indexes on expressions, which (in my understanding) should allow you to define indexes on your JSON data as well. PostgreSQL has some...

Combining tail && journalctl

You could use: journalctl -u service-name -f -f, --follow Show only the most recent journal entries, and continuously print new entries as they are appended to the journal. Here I've added "service-name" to distinguish this answer from others; you substitute...

how can shellshock be exploited over SSH?

One example where this can be exploited is on servers with an authorized_keys forced command. When adding an entry to ~/.ssh/authorized_keys, you can prefix the line with command="foo" to force foo to be run any time that ssh public key is used. With this...

Why doesn’t the tilde (~) expand inside double quotes?

The reason, because inside double quotes, tilde ~ has no special meaning, it's treated as literal. POSIX defines Double-Quotes as: Enclosing characters in double-quotes ( "" ) shall preserve the literal value of all characters within the double-quotes, with the...

What is GNU Info for?

GNU Info was designed to offer documentation that was comprehensive, hyperlinked, and possible to output to multiple formats. Man pages were available, and they were great at providing printed output. However, they were designed such that each man page had a...

Set systemd service to execute after fstab mount

a CIFS network location is mounted via /etc/fstab to /mnt/ on boot-up. No, it is not. Get this right, and the rest falls into place naturally. The mount is handled by a (generated) systemd mount unit that will be named something like mnt-wibble.mount. You can...

Merge two video clips into one, placing them next to each other

To be honest, using the accepted answer resulted in a lot of dropped frames for me. However, using the hstack filter_complex produced perfectly fluid output: ffmpeg -i left.mp4 -i right.mp4 -filter_complex hstack output.mp4 ffmpeg -i input1.mp4 -i input2.mp4...

How portable are /dev/stdin, /dev/stdout and /dev/stderr?

It's been available on Linux back into its prehistory. It is not POSIX, although many actual shells (including AT&T ksh and bash) will simulate it if it's not present in the OS; note that this simulation only works at the shell level (i.e. redirection or...

How can I increase the number of inodes in an ext4 filesystem?

It seems that you have a lot more files than normal expectation. I don't know whether there is a solution to change the inode table size dynamically. I'm afraid that you need to back-up your data, and create new filesystem, and restore your data. To create new...

Why doesn’t cp have a progress bar like wget?

The tradition in unix tools is to display messages only if something goes wrong. I think this is both for design and practical reasons. The design is intended to make it obvious when something goes wrong: you get an error message, and it's not drowned in...