| Message ID | 56DD5B26.30902@suse.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Mon, 7 Mar 2016, Owen Synge wrote: > Dear all, > > I have a simple idea that I think would be beneficial for ceph's > services. I would be curious to hear others opinions. > > At the moment the only way (I know) to configure specific entities in > ceph such as a specific port for a mds or rgw is via ${CLUSTERNAME}.conf > which is usually set to ceph.conf. For brevity I will call this file > ceph.conf. In jewel there is a new path added to the config search path: /var/lib/ceph/$type/$cluster-$id/config. This is checked before /etc/ceph/$cluster.conf, which means that you have to put *all* config values in the new location if you use it, but it might capture your use-case? If you need both config paths to work, one idea is to add 'include <path>' support to the conf parser... hoping to avoid making things more complicated if we don't have a strong reason for it, though! > The current setup leads to complexities as ideally I want all ceph.conf > files to be identical across a cluster, and at the same time, I want to > add services such as rgw or mds without changing ceph.conf. I also want > to template a ceph.conf cluster. > > If we could always get these settings via environment variables, it > would make setting up ceph with tools such as cluster/config management > tools such as puppet, chef, and salt easier. > > So here is the diff of the systemd service files I am thinking of: > > # diff -u ceph-mds@.service /tmp/ceph-mds@.service > --- ceph-mds@.service 2016-03-01 20:48:05.000000000 -0500 > +++ /tmp/ceph-mds@.service 2016-03-07 05:14:43.392000000 -0500 > @@ -8,8 +8,8 @@ > LimitNOFILE=1048576 > LimitNPROC=1048576 > EnvironmentFile=-/etc/sysconfig/ceph > -Environment=CLUSTER=ceph > -ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} --id %i --setuser > ceph --setgroup ceph > +EnvironmentFile=-/var/lib/ceph/env/mds/%i > +ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} --id %i --setuser > ceph --setgroup ceph -m ${BIND_IPV4}:${BIND_PORT} > ExecReload=/bin/kill -HUP $MAINPID > PrivateDevices=yes > ProtectHome=true > > First you should note the extra EnvironmentFile line: > > EnvironmentFile=-/var/lib/ceph/mds/systemd/%i > > This means that if the mds service is started with parameter "param_one" > then the environment will be set with the content of > > /var/lib/ceph/env/mds/param_one > > The "-" means don’t error if the above file does not exist. > > Hence we could add service instance variables such as ${BIND_IPV4} and > ${BIND_PORT} as well as important values like ${CLUSTER} to values like > ceph. > > Having played with systemd, it seems that environment variables are > updated by the last assignment in systemd just like shell. > > I have *not* found a good way to put in conditionals in such systemd > "EnvironmentFile" content. Hence it would be impossible to nicely > default command line parameters without using dirty tricks. Specifically > what we cant do in the systemd "EnvironmentFile" context: > > if [ X"${BIND_IPV4}${BIND_PORT}" != X ] ; then > EXTRA_ARGS="-m ${BIND_IPV4}:${BIND_PORT}" > fi > > Hence I think we have to accept to integrate with systemd nicely we > should allow setting to be read from environment rather than only from > the command line. > > What do people think? I would probably make it '/var/lib/ceph/$type/$cluster-$id/env'. You're sure systemd lets you specify multiple environment files? We can't/shouldn't stop looking at the standard location... sage
On 03/07/2016 12:47 PM, Sage Weil wrote: > On Mon, 7 Mar 2016, Owen Synge wrote: >> Dear all, >> >> I have a simple idea that I think would be beneficial for ceph's >> services. I would be curious to hear others opinions. >> >> At the moment the only way (I know) to configure specific entities in >> ceph such as a specific port for a mds or rgw is via ${CLUSTERNAME}.conf >> which is usually set to ceph.conf. For brevity I will call this file >> ceph.conf. > > In jewel there is a new path added to the config search path: > /var/lib/ceph/$type/$cluster-$id/config. This is checked before > /etc/ceph/$cluster.conf, which means that you have to put *all* config > values in the new location if you use it, but it might capture your > use-case? It does capture it nearly 100%. While it does capture my usecase I still suggest we go further and have all variables for an entity able to come from its own local environment, rather than just sections in the config search. The (current list of) reasons: (1) environment variables are easier to pass in systemd format than a ceph.conf file for cluster management tools. (2) We can support much richer overloading and defaulting with environment variables. Allowing for example defaulting cluster name as "ceph" and overriding it on a per module basis, this can be extended to other variables. This said, since your solution does cover most of the test cases (excluding cluster name) The importance of this change is greatly reduced. > If you need both config paths to work, one idea is to add 'include <path>' > support to the conf parser... hoping to avoid making things more > complicated if we don't have a strong reason for it, though! Thanks for the hint, I dont feel this is needed yet, and would rather avoid complications caused by such an idea unless I had too. >> The current setup leads to complexities as ideally I want all ceph.conf >> files to be identical across a cluster, and at the same time, I want to >> add services such as rgw or mds without changing ceph.conf. I also want >> to template a ceph.conf cluster. >> >> If we could always get these settings via environment variables, it >> would make setting up ceph with tools such as cluster/config management >> tools such as puppet, chef, and salt easier. >> >> So here is the diff of the systemd service files I am thinking of: >> >> # diff -u ceph-mds@.service /tmp/ceph-mds@.service >> --- ceph-mds@.service 2016-03-01 20:48:05.000000000 -0500 >> +++ /tmp/ceph-mds@.service 2016-03-07 05:14:43.392000000 -0500 >> @@ -8,8 +8,8 @@ >> LimitNOFILE=1048576 >> LimitNPROC=1048576 >> EnvironmentFile=-/etc/sysconfig/ceph >> -Environment=CLUSTER=ceph >> -ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} --id %i --setuser >> ceph --setgroup ceph >> +EnvironmentFile=-/var/lib/ceph/env/mds/%i >> +ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} --id %i --setuser >> ceph --setgroup ceph -m ${BIND_IPV4}:${BIND_PORT} >> ExecReload=/bin/kill -HUP $MAINPID >> PrivateDevices=yes >> ProtectHome=true >> >> First you should note the extra EnvironmentFile line: >> >> EnvironmentFile=-/var/lib/ceph/mds/systemd/%i >> >> This means that if the mds service is started with parameter "param_one" >> then the environment will be set with the content of >> >> /var/lib/ceph/env/mds/param_one >> >> The "-" means don’t error if the above file does not exist. >> >> Hence we could add service instance variables such as ${BIND_IPV4} and >> ${BIND_PORT} as well as important values like ${CLUSTER} to values like >> ceph. >> >> Having played with systemd, it seems that environment variables are >> updated by the last assignment in systemd just like shell. >> >> I have *not* found a good way to put in conditionals in such systemd >> "EnvironmentFile" content. Hence it would be impossible to nicely >> default command line parameters without using dirty tricks. Specifically >> what we cant do in the systemd "EnvironmentFile" context: >> >> if [ X"${BIND_IPV4}${BIND_PORT}" != X ] ; then >> EXTRA_ARGS="-m ${BIND_IPV4}:${BIND_PORT}" >> fi >> >> Hence I think we have to accept to integrate with systemd nicely we >> should allow setting to be read from environment rather than only from >> the command line. >> >> What do people think? > > I would probably make it '/var/lib/ceph/$type/$cluster-$id/env'. > > You're sure systemd lets you specify multiple environment files? Many examples of doing this are available. Also I tested it and it worked on Friday on SUSE LEAP for my simple test case. We have more cases I could test though. > We > can't/shouldn't stop looking at the standard location... Totally agree. Best regards Owen -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
--- ceph-mds@.service 2016-03-01 20:48:05.000000000 -0500 +++ /tmp/ceph-mds@.service 2016-03-07 05:14:43.392000000 -0500 @@ -8,8 +8,8 @@ LimitNOFILE=1048576 LimitNPROC=1048576 EnvironmentFile=-/etc/sysconfig/ceph -Environment=CLUSTER=ceph -ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph +EnvironmentFile=-/var/lib/ceph/env/mds/%i +ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph -m ${BIND_IPV4}:${BIND_PORT} ExecReload=/bin/kill -HUP $MAINPID
Dear all, I have a simple idea that I think would be beneficial for ceph's services. I would be curious to hear others opinions. At the moment the only way (I know) to configure specific entities in ceph such as a specific port for a mds or rgw is via ${CLUSTERNAME}.conf which is usually set to ceph.conf. For brevity I will call this file ceph.conf. The current setup leads to complexities as ideally I want all ceph.conf files to be identical across a cluster, and at the same time, I want to add services such as rgw or mds without changing ceph.conf. I also want to template a ceph.conf cluster. If we could always get these settings via environment variables, it would make setting up ceph with tools such as cluster/config management tools such as puppet, chef, and salt easier. So here is the diff of the systemd service files I am thinking of: # diff -u ceph-mds@.service /tmp/ceph-mds@.service PrivateDevices=yes ProtectHome=true First you should note the extra EnvironmentFile line: EnvironmentFile=-/var/lib/ceph/mds/systemd/%i This means that if the mds service is started with parameter "param_one" then the environment will be set with the content of /var/lib/ceph/env/mds/param_one The "-" means don’t error if the above file does not exist. Hence we could add service instance variables such as ${BIND_IPV4} and ${BIND_PORT} as well as important values like ${CLUSTER} to values like ceph. Having played with systemd, it seems that environment variables are updated by the last assignment in systemd just like shell. I have *not* found a good way to put in conditionals in such systemd "EnvironmentFile" content. Hence it would be impossible to nicely default command line parameters without using dirty tricks. Specifically what we cant do in the systemd "EnvironmentFile" context: if [ X"${BIND_IPV4}${BIND_PORT}" != X ] ; then EXTRA_ARGS="-m ${BIND_IPV4}:${BIND_PORT}" fi Hence I think we have to accept to integrate with systemd nicely we should allow setting to be read from environment rather than only from the command line. What do people think? Best wishes Owen -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html