| Message ID | CA+Hcz5-uxiNMQiNXKfbETiQG0D6-nUrbV5oGpmMqn8oWi4DwGw@mail.gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
Hi, On Wed, Aug 21, 2013 at 10:01 PM, Noah Watkins <noah.watkins@inktank.com> wrote: > This patch will turn on the option (which should also fix your problem > if I understand correctly?), and should probably be committed anyway > as newer versions of autotools will complain loudly about our current > Makefile structure. The patch is indeed quite trivial, and I already enabled it for my own dev branches, but I like it to be upstream or else my code could never be pulled :-) > I believe it is just historical, and unfortunately has just been > repeated over and over. Ideally I think that there should be a > restructuring to place a Makefile.am in every subdirectory. This would > address your issue and make it significantly easier to deal with > situations where we want to build a subset of Ceph, such as just FUSE > and librados, for example. I agree. Is there already a project running for this? (I feel like I'm hitting depedency after dependency... I started out trying to implement RGW S3 Keystone support, for that I noticed that plugins would be really nice, and now I'm hitting automake issues :-)... Mayby someday I'll finish something useful hehe) Roald -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 21 Aug 2013, Noah Watkins wrote: > On Wed, Aug 21, 2013 at 12:45 PM, Roald van Loon <roaldvanloon@gmail.com> wrote: > > > > from auto-registering the plugins in the RGW core. The only fix for > > this is making the RGW core aware of the subdirs/plugins, but I think > > that's nasty design. I'd like to have it in my make conf. > > This patch will turn on the option (which should also fix your problem > if I understand correctly?), and should probably be committed anyway > as newer versions of autotools will complain loudly about our current > Makefile structure. > > diff --git a/src/Makefile.am b/src/Makefile.am > index 93f3331..fb7c9dd 100644 > --- a/src/Makefile.am > +++ b/src/Makefile.am > @@ -1,4 +1,4 @@ > -AUTOMAKE_OPTIONS = gnu > +AUTOMAKE_OPTIONS = gnu subdir-objects > SUBDIRS = ocf java > DIST_SUBDIRS = gtest ocf libs3 java > > > So, the question is; is there a reason why we don't use subdir objects? > > I believe it is just historical, and unfortunately has just been > repeated over and over. Ideally I think that there should be a > restructuring to place a Makefile.am in every subdirectory. This would > address your issue and make it significantly easier to deal with > situations where we want to build a subset of Ceph, such as just FUSE > and librados, for example. Yes, the Makefile.am is in dire need of from TLC from someone who knows a bit of autotools-fu. It is only this way because in the beginning I didn't know any better. sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Aug 21, 2013 at 10:41 PM, Sage Weil <sage@inktank.com> wrote: > Yes, the Makefile.am is in dire need of from TLC from someone who knows a > bit of autotools-fu. It is only this way because in the beginning I > didn't know any better. Well, my average knowledge of autotools could at least fix this particular issue and clean up a bit more. It's a start I guess and helps me to continue my RGW things. I'll send out a pull request when I've found some time to implement and test this. Roald -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Roald, Sage just pointed me at your wip-automake branch. I also just pushed up a branch, make-refactor, that I was hacking on a bit. Not sure how much overlap there is, or if my approach is bogus, but I thought I'd point it out to see if there is anything that can be combined :) -Noah On Wed, Aug 21, 2013 at 2:01 PM, Roald van Loon <roaldvanloon@gmail.com> wrote: > On Wed, Aug 21, 2013 at 10:41 PM, Sage Weil <sage@inktank.com> wrote: >> Yes, the Makefile.am is in dire need of from TLC from someone who knows a >> bit of autotools-fu. It is only this way because in the beginning I >> didn't know any better. > > Well, my average knowledge of autotools could at least fix this > particular issue and clean up a bit more. It's a start I guess and > helps me to continue my RGW things. > > I'll send out a pull request when I've found some time to implement > and test this. > > Roald -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Noah, I just had a quick look at your build-refactor branch, and I think the greatest difference is that you use recursive builds and I don't. I'm more in favor of non-recursive builds using includes for a number of reasons. I think the most important reasons for me are; 1) recursive make leads to repetitive AM code 2) recursive make takes much more time to compile (as each directory needs to run configure and probably most important: you loose optimal -jX usage due to serialization) 3) non-recursive make knows all deps so rebuildings is much quicker (it only compiles/links what is required instead of entering all subdirs) There is ÏMHO one good reason to use recursive build, and that is separation of AM code. However, that can be easily achieved with includes and subdir-objects. I think this is the most important difference between your and my approach, and I like to hear your arguments for recursive builds so we can agree on recursive vs non-recursive make. Then I think it would be great to combine work! Roald On Fri, Sep 6, 2013 at 7:27 PM, Noah Watkins <noah.watkins@inktank.com> wrote: > Hi Roald, > > Sage just pointed me at your wip-automake branch. I also just pushed > up a branch, make-refactor, that I was hacking on a bit. Not sure how > much overlap there is, or if my approach is bogus, but I thought I'd > point it out to see if there is anything that can be combined :) > > -Noah > > On Wed, Aug 21, 2013 at 2:01 PM, Roald van Loon <roaldvanloon@gmail.com> wrote: >> On Wed, Aug 21, 2013 at 10:41 PM, Sage Weil <sage@inktank.com> wrote: >>> Yes, the Makefile.am is in dire need of from TLC from someone who knows a >>> bit of autotools-fu. It is only this way because in the beginning I >>> didn't know any better. >> >> Well, my average knowledge of autotools could at least fix this >> particular issue and clean up a bit more. It's a start I guess and >> helps me to continue my RGW things. >> >> I'll send out a pull request when I've found some time to implement >> and test this. >> >> Roald -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
The non-recursive approach is interesting. I just had a quick look in the tree I despise building the most, openmpi. It has 414 Makefile.am, and uses recursive builds. The rebuild definitely takes a while to visit all the sub-dirs, and is pretty annoying when my patience is low :) And there is definitely a big +1 for avoiding the SUBDIRS synchronization that slows down parallel make. I think the benefits of using recursive builds are that it may be familiar to the most people, it reflects the methods/suggestions in the automake manual, and, most importantly, it would seem that its use forces good decomposition where as a non-recursive approach relies on guidelines that are easily broken. Given that the Ceph tree is relatively small (certinaly in comparison to the 414 directory openmpi monster), are there benefits to the non-recursive approach that are not performance related? - Noah On Sat, Sep 7, 2013 at 1:52 AM, Roald van Loon <roaldvanloon@gmail.com> wrote: > Hi Noah, > > I just had a quick look at your build-refactor branch, and I think the > greatest difference is that you use recursive builds and I don't. I'm > more in favor of non-recursive builds using includes for a number of > reasons. I think the most important reasons for me are; > > 1) recursive make leads to repetitive AM code > 2) recursive make takes much more time to compile (as each directory > needs to run configure and probably most important: you loose optimal > -jX usage due to serialization) > 3) non-recursive make knows all deps so rebuildings is much quicker > (it only compiles/links what is required instead of entering all > subdirs) > > There is ÏMHO one good reason to use recursive build, and that is > separation of AM code. However, that can be easily achieved with > includes and subdir-objects. > > I think this is the most important difference between your and my > approach, and I like to hear your arguments for recursive builds so we > can agree on recursive vs non-recursive make. Then I think it would be > great to combine work! > > Roald > > On Fri, Sep 6, 2013 at 7:27 PM, Noah Watkins <noah.watkins@inktank.com> wrote: >> Hi Roald, >> >> Sage just pointed me at your wip-automake branch. I also just pushed >> up a branch, make-refactor, that I was hacking on a bit. Not sure how >> much overlap there is, or if my approach is bogus, but I thought I'd >> point it out to see if there is anything that can be combined :) >> >> -Noah >> >> On Wed, Aug 21, 2013 at 2:01 PM, Roald van Loon <roaldvanloon@gmail.com> wrote: >>> On Wed, Aug 21, 2013 at 10:41 PM, Sage Weil <sage@inktank.com> wrote: >>>> Yes, the Makefile.am is in dire need of from TLC from someone who knows a >>>> bit of autotools-fu. It is only this way because in the beginning I >>>> didn't know any better. >>> >>> Well, my average knowledge of autotools could at least fix this >>> particular issue and clean up a bit more. It's a start I guess and >>> helps me to continue my RGW things. >>> >>> I'll send out a pull request when I've found some time to implement >>> and test this. >>> >>> Roald -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Oh, and one question about the non-recursive approach. If I stick a Makefile.am in the test/ directory I can do things like: LDADD = all-the-test-dependencies and then avoid redundant per-target primaries like test_LDADD = (deps) $(LDADD), because it applies to everything in the Makefile. Is that possible with the include approach, or would a naked LDADD in an included Makefile fragment affect all the targets in the file including it? -Noah On Sat, Sep 7, 2013 at 10:38 AM, Noah Watkins <noah.watkins@inktank.com> wrote: > The non-recursive approach is interesting. I just had a quick look in > the tree I despise building the most, openmpi. It has 414 Makefile.am, > and uses recursive builds. The rebuild definitely takes a while to > visit all the sub-dirs, and is pretty annoying when my patience is low > :) > > And there is definitely a big +1 for avoiding the SUBDIRS > synchronization that slows down parallel make. > > I think the benefits of using recursive builds are that it may be > familiar to the most people, it reflects the methods/suggestions in > the automake manual, and, most importantly, it would seem that its use > forces good decomposition where as a non-recursive approach relies on > guidelines that are easily broken. > > Given that the Ceph tree is relatively small (certinaly in comparison > to the 414 directory openmpi monster), are there benefits to the > non-recursive approach that are not performance related? > > - Noah > > On Sat, Sep 7, 2013 at 1:52 AM, Roald van Loon <roaldvanloon@gmail.com> wrote: >> Hi Noah, >> >> I just had a quick look at your build-refactor branch, and I think the >> greatest difference is that you use recursive builds and I don't. I'm >> more in favor of non-recursive builds using includes for a number of >> reasons. I think the most important reasons for me are; >> >> 1) recursive make leads to repetitive AM code >> 2) recursive make takes much more time to compile (as each directory >> needs to run configure and probably most important: you loose optimal >> -jX usage due to serialization) >> 3) non-recursive make knows all deps so rebuildings is much quicker >> (it only compiles/links what is required instead of entering all >> subdirs) >> >> There is ÏMHO one good reason to use recursive build, and that is >> separation of AM code. However, that can be easily achieved with >> includes and subdir-objects. >> >> I think this is the most important difference between your and my >> approach, and I like to hear your arguments for recursive builds so we >> can agree on recursive vs non-recursive make. Then I think it would be >> great to combine work! >> >> Roald >> >> On Fri, Sep 6, 2013 at 7:27 PM, Noah Watkins <noah.watkins@inktank.com> wrote: >>> Hi Roald, >>> >>> Sage just pointed me at your wip-automake branch. I also just pushed >>> up a branch, make-refactor, that I was hacking on a bit. Not sure how >>> much overlap there is, or if my approach is bogus, but I thought I'd >>> point it out to see if there is anything that can be combined :) >>> >>> -Noah >>> >>> On Wed, Aug 21, 2013 at 2:01 PM, Roald van Loon <roaldvanloon@gmail.com> wrote: >>>> On Wed, Aug 21, 2013 at 10:41 PM, Sage Weil <sage@inktank.com> wrote: >>>>> Yes, the Makefile.am is in dire need of from TLC from someone who knows a >>>>> bit of autotools-fu. It is only this way because in the beginning I >>>>> didn't know any better. >>>> >>>> Well, my average knowledge of autotools could at least fix this >>>> particular issue and clean up a bit more. It's a start I guess and >>>> helps me to continue my RGW things. >>>> >>>> I'll send out a pull request when I've found some time to implement >>>> and test this. >>>> >>>> Roald -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, Sep 7, 2013 at 7:47 PM, Noah Watkins <noah.watkins@inktank.com> wrote: > Oh, and one question about the non-recursive approach. If I stick a > Makefile.am in the test/ directory I can do things like: > > LDADD = all-the-test-dependencies > > and then avoid redundant per-target primaries like test_LDADD = (deps) > $(LDADD), because it applies to everything in the Makefile. > > Is that possible with the include approach, or would a naked LDADD in > an included Makefile fragment affect all the targets in the file > including it? LDADD = xyz would indeed affect all targets. However, it's not something you want to do anyway; using an _LDADD at a target is less confusing and less prone to errors because you know exactly what libraries a target needs. For instance, in test/Makefile.am you can have a debug target depending on libglobal, which has dependencies set by libglobal itself; CEPH_GLOBAL = $(LIBGLOBAL) $(PTHREAD_LIBS) -lm $(CRYPTO_LIBS) $(EXTRALIBS) And then in test/Makefile.am; ceph_test_crypto_SOURCES = test/testcrypto.cc ceph_test_crypto_LDADD = $(CEPH_GLOBAL) bin_DEBUGPROGRAMS += ceph_test_crypto And a unittest also depending on libosd; unittest_pglog_SOURCES = test/osd/TestPGLog.cc unittest_pglog_LDADD = $(LIBOSD) $(CEPH_GLOBAL) check_PROGRAMS += unittest_pglog However, libosd requires libos and libosdc, but that dependency is set by libosd; LIBOSD += $(LIBOSDC) $(LIBOS) This way, you have the dependencies in the right place. With recusive builds you'll need an "LDADD = libosd.la libosdc.la libos.la libglobal.la $(PTHREAD_LIBS) -lm $(CRYPTO_LIBS) $(EXTRALIBS)", so basically you're setting the dependencies of the required libraries in the makefile requiring those libraries, which is IMHO way to complex. >> I think the benefits of using recursive builds are that it may be >> familiar to the most people, it reflects the methods/suggestions in >> the automake manual, and, most importantly, it would seem that its use >> forces good decomposition where as a non-recursive approach relies on >> guidelines that are easily broken. I don't know how which method is more familiar, but I personally think that anyone understanding recursive automake is capable of understanding a simple include :-) The decomposition is a valid argument. I think that there are some libraries which might benefit from complete separation, like librados and a "libceph_client" or something like it. Those can be separated, but most other can't. The mon, mds, os, and osd subdirs have inter-dependencies for instance. We might need to restructure the source tree anyway because at some points it has grown messy (for instance, libcommon including stuff from mds, mon but also from include). However, I think implementing a recursive automake right now forces us to do two things at once; cleanup the makefiles and do some restructuring in the subdirs. I personally think it's best to start with cleaning up makefiles and use an include per subdir, so we can restructure the subdirs into segregated libraries later on. So I all boils down to; what to do first :-) Because I agree some things are better of with recursive builds, but it might be wise not to do that before we have revisited the source tree layout. Roald -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
I'm particularly keen on doing what we can to reduce build time. I also agree there is a fair bit of tree restructuring that needs to be done. (For example, libcommon.la is huge and includes lots of stuff that most targets don't actually need; breaking this into smaller pieces would probably help.) Anyway, my gut says that starting with the includes seems like an easy incremental path to get us there. Then we can break out any fully modular components where it makes sense... sage On Sat, 7 Sep 2013, Roald van Loon wrote: > On Sat, Sep 7, 2013 at 7:47 PM, Noah Watkins <noah.watkins@inktank.com> wrote: > > Oh, and one question about the non-recursive approach. If I stick a > > Makefile.am in the test/ directory I can do things like: > > > > LDADD = all-the-test-dependencies > > > > and then avoid redundant per-target primaries like test_LDADD = (deps) > > $(LDADD), because it applies to everything in the Makefile. > > > > Is that possible with the include approach, or would a naked LDADD in > > an included Makefile fragment affect all the targets in the file > > including it? > > LDADD = xyz would indeed affect all targets. However, it's not > something you want to do anyway; using an _LDADD at a target is less > confusing and less prone to errors because you know exactly what > libraries a target needs. > > For instance, in test/Makefile.am you can have a debug target > depending on libglobal, which has dependencies set by libglobal > itself; > > CEPH_GLOBAL = $(LIBGLOBAL) $(PTHREAD_LIBS) -lm $(CRYPTO_LIBS) $(EXTRALIBS) > > And then in test/Makefile.am; > > ceph_test_crypto_SOURCES = test/testcrypto.cc > ceph_test_crypto_LDADD = $(CEPH_GLOBAL) > bin_DEBUGPROGRAMS += ceph_test_crypto > > And a unittest also depending on libosd; > > unittest_pglog_SOURCES = test/osd/TestPGLog.cc > unittest_pglog_LDADD = $(LIBOSD) $(CEPH_GLOBAL) > check_PROGRAMS += unittest_pglog > > However, libosd requires libos and libosdc, but that dependency is set > by libosd; > > LIBOSD += $(LIBOSDC) $(LIBOS) > > This way, you have the dependencies in the right place. With recusive > builds you'll need an "LDADD = libosd.la libosdc.la libos.la > libglobal.la $(PTHREAD_LIBS) -lm $(CRYPTO_LIBS) $(EXTRALIBS)", so > basically you're setting the dependencies of the required libraries in > the makefile requiring those libraries, which is IMHO way to complex. > > >> I think the benefits of using recursive builds are that it may be > >> familiar to the most people, it reflects the methods/suggestions in > >> the automake manual, and, most importantly, it would seem that its use > >> forces good decomposition where as a non-recursive approach relies on > >> guidelines that are easily broken. > > I don't know how which method is more familiar, but I personally think > that anyone understanding recursive automake is capable of > understanding a simple include :-) > > The decomposition is a valid argument. I think that there are some > libraries which might benefit from complete separation, like librados > and a "libceph_client" or something like it. Those can be separated, > but most other can't. The mon, mds, os, and osd subdirs have > inter-dependencies for instance. > > We might need to restructure the source tree anyway because at some > points it has grown messy (for instance, libcommon including stuff > from mds, mon but also from include). However, I think implementing a > recursive automake right now forces us to do two things at once; > cleanup the makefiles and do some restructuring in the subdirs. I > personally think it's best to start with cleaning up makefiles and use > an include per subdir, so we can restructure the subdirs into > segregated libraries later on. > > So I all boils down to; what to do first :-) Because I agree some > things are better of with recursive builds, but it might be wise not > to do that before we have revisited the source tree layout. > > Roald > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
I'm so excited to have a refactored automake setup :) I was just looking through build-refactor, and it doesn't really look like there is much that could be reused for the non-recrusive approach. I'll leave it up for a few days, just in case. On Sat, Sep 7, 2013 at 1:11 PM, Roald van Loon <roaldvanloon@gmail.com> wrote: > On Sat, Sep 7, 2013 at 7:47 PM, Noah Watkins <noah.watkins@inktank.com> wrote: >> Oh, and one question about the non-recursive approach. If I stick a >> Makefile.am in the test/ directory I can do things like: >> >> LDADD = all-the-test-dependencies >> >> and then avoid redundant per-target primaries like test_LDADD = (deps) >> $(LDADD), because it applies to everything in the Makefile. >> >> Is that possible with the include approach, or would a naked LDADD in >> an included Makefile fragment affect all the targets in the file >> including it? > > LDADD = xyz would indeed affect all targets. However, it's not > something you want to do anyway; using an _LDADD at a target is less > confusing and less prone to errors because you know exactly what > libraries a target needs. > > For instance, in test/Makefile.am you can have a debug target > depending on libglobal, which has dependencies set by libglobal > itself; > > CEPH_GLOBAL = $(LIBGLOBAL) $(PTHREAD_LIBS) -lm $(CRYPTO_LIBS) $(EXTRALIBS) > > And then in test/Makefile.am; > > ceph_test_crypto_SOURCES = test/testcrypto.cc > ceph_test_crypto_LDADD = $(CEPH_GLOBAL) > bin_DEBUGPROGRAMS += ceph_test_crypto > > And a unittest also depending on libosd; > > unittest_pglog_SOURCES = test/osd/TestPGLog.cc > unittest_pglog_LDADD = $(LIBOSD) $(CEPH_GLOBAL) > check_PROGRAMS += unittest_pglog > > However, libosd requires libos and libosdc, but that dependency is set > by libosd; > > LIBOSD += $(LIBOSDC) $(LIBOS) > > This way, you have the dependencies in the right place. With recusive > builds you'll need an "LDADD = libosd.la libosdc.la libos.la > libglobal.la $(PTHREAD_LIBS) -lm $(CRYPTO_LIBS) $(EXTRALIBS)", so > basically you're setting the dependencies of the required libraries in > the makefile requiring those libraries, which is IMHO way to complex. > >>> I think the benefits of using recursive builds are that it may be >>> familiar to the most people, it reflects the methods/suggestions in >>> the automake manual, and, most importantly, it would seem that its use >>> forces good decomposition where as a non-recursive approach relies on >>> guidelines that are easily broken. > > I don't know how which method is more familiar, but I personally think > that anyone understanding recursive automake is capable of > understanding a simple include :-) > > The decomposition is a valid argument. I think that there are some > libraries which might benefit from complete separation, like librados > and a "libceph_client" or something like it. Those can be separated, > but most other can't. The mon, mds, os, and osd subdirs have > inter-dependencies for instance. > > We might need to restructure the source tree anyway because at some > points it has grown messy (for instance, libcommon including stuff > from mds, mon but also from include). However, I think implementing a > recursive automake right now forces us to do two things at once; > cleanup the makefiles and do some restructuring in the subdirs. I > personally think it's best to start with cleaning up makefiles and use > an include per subdir, so we can restructure the subdirs into > segregated libraries later on. > > So I all boils down to; what to do first :-) Because I agree some > things are better of with recursive builds, but it might be wise not > to do that before we have revisited the source tree layout. > > Roald -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Noah, Sage, The build time has improved in wip-automake somewhat (mainly because almost all files are now only compiled to objects once instead of per-target), but much of the time goes into linking. > I was just looking through build-refactor, and it doesn't really look > like there is much that could be reused for the non-recrusive > approach. I'll leave it up for a few days, just in case. It would be great if you could look at wip-automake and see if you have ideas/improvements or stuff you implemented that I didn't. All non-automake changes that I needed to make this branch work just got merged to master, so this branch is now only about changes .am files. Of course every src/Makefile.am adjustment in master will require a rebase, so this branch is a real work-in-progress and gets force-pushed regularly :-) Roald -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Thanks for doing this. Also in the same interest of building Ceph faster, we've been experimenting with a CMake translation of the current AM build. I have no idea yet if it will be any help ;). Matt ----- "Noah Watkins" <noah.watkins@inktank.com> wrote: > I'm so excited to have a refactored automake setup :) > > I was just looking through build-refactor, and it doesn't really look > like there is much that could be reused for the non-recrusive > approach. I'll leave it up for a few days, just in case. > > On Sat, Sep 7, 2013 at 1:11 PM, Roald van Loon > <roaldvanloon@gmail.com> wrote: > > On Sat, Sep 7, 2013 at 7:47 PM, Noah Watkins > <noah.watkins@inktank.com> wrote: > >> Oh, and one question about the non-recursive approach. If I stick > a > >> Makefile.am in the test/ directory I can do things like: > >> > >> LDADD = all-the-test-dependencies > >> > >> and then avoid redundant per-target primaries like test_LDADD = > (deps) > >> $(LDADD), because it applies to everything in the Makefile. > >> > >> Is that possible with the include approach, or would a naked LDADD > in > >> an included Makefile fragment affect all the targets in the file > >> including it? > > > > LDADD = xyz would indeed affect all targets. However, it's not > > something you want to do anyway; using an _LDADD at a target is > less > > confusing and less prone to errors because you know exactly what > > libraries a target needs. > > > > For instance, in test/Makefile.am you can have a debug target > > depending on libglobal, which has dependencies set by libglobal > > itself; > > > > CEPH_GLOBAL = $(LIBGLOBAL) $(PTHREAD_LIBS) -lm $(CRYPTO_LIBS) > $(EXTRALIBS) > > > > And then in test/Makefile.am; > > > > ceph_test_crypto_SOURCES = test/testcrypto.cc > > ceph_test_crypto_LDADD = $(CEPH_GLOBAL) > > bin_DEBUGPROGRAMS += ceph_test_crypto > > > > And a unittest also depending on libosd; > > > > unittest_pglog_SOURCES = test/osd/TestPGLog.cc > > unittest_pglog_LDADD = $(LIBOSD) $(CEPH_GLOBAL) > > check_PROGRAMS += unittest_pglog > > > > However, libosd requires libos and libosdc, but that dependency is > set > > by libosd; > > > > LIBOSD += $(LIBOSDC) $(LIBOS) > > > > This way, you have the dependencies in the right place. With > recusive > > builds you'll need an "LDADD = libosd.la libosdc.la libos.la > > libglobal.la $(PTHREAD_LIBS) -lm $(CRYPTO_LIBS) $(EXTRALIBS)", so > > basically you're setting the dependencies of the required libraries > in > > the makefile requiring those libraries, which is IMHO way to > complex. > > > >>> I think the benefits of using recursive builds are that it may be > >>> familiar to the most people, it reflects the methods/suggestions > in > >>> the automake manual, and, most importantly, it would seem that its > use > >>> forces good decomposition where as a non-recursive approach relies > on > >>> guidelines that are easily broken. > > > > I don't know how which method is more familiar, but I personally > think > > that anyone understanding recursive automake is capable of > > understanding a simple include :-) > > > > The decomposition is a valid argument. I think that there are some > > libraries which might benefit from complete separation, like > librados > > and a "libceph_client" or something like it. Those can be > separated, > > but most other can't. The mon, mds, os, and osd subdirs have > > inter-dependencies for instance. > > > > We might need to restructure the source tree anyway because at some > > points it has grown messy (for instance, libcommon including stuff > > from mds, mon but also from include). However, I think implementing > a > > recursive automake right now forces us to do two things at once; > > cleanup the makefiles and do some restructuring in the subdirs. I > > personally think it's best to start with cleaning up makefiles and > use > > an include per subdir, so we can restructure the subdirs into > > segregated libraries later on. > > > > So I all boils down to; what to do first :-) Because I agree some > > things are better of with recursive builds, but it might be wise > not > > to do that before we have revisited the source tree layout. > > > > Roald > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, 8 Sep 2013, Roald van Loon wrote: > Hi Noah, Sage, > > The build time has improved in wip-automake somewhat (mainly because > almost all files are now only compiled to objects once instead of > per-target), but much of the time goes into linking. > > > I was just looking through build-refactor, and it doesn't really look > > like there is much that could be reused for the non-recrusive > > approach. I'll leave it up for a few days, just in case. > > It would be great if you could look at wip-automake and see if you > have ideas/improvements or stuff you implemented that I didn't. All > non-automake changes that I needed to make this branch work just got > merged to master, so this branch is now only about changes .am files. > Of course every src/Makefile.am adjustment in master will require a > rebase, so this branch is a real work-in-progress and gets > force-pushed regularly :-) I took a quick look and it looks pretty great. The builds are coming back okay, too.. any compelling reason not to merge it now? BTW, Roald, I think the obj_bencher cct change did something strange... see http://tracker.ceph.com/issues/6256. Trying to reproduce it now. sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> The build time has improved in wip-automake somewhat (mainly because > almost all files are now only compiled to objects once instead of > per-target), but much of the time goes into linking. Roald, Are you using gold as a linker already instead of the traditional binutils bdf based ld. It does make a difference esp on large C++ code bases (where linking takes a lot of time). - M >> I was just looking through build-refactor, and it doesn't really look >> like there is much that could be reused for the non-recrusive >> approach. I'll leave it up for a few days, just in case. > > It would be great if you could look at wip-automake and see if you > have ideas/improvements or stuff you implemented that I didn't. All > non-automake changes that I needed to make this branch work just got > merged to master, so this branch is now only about changes .am files. > Of course every src/Makefile.am adjustment in master will require a > rebase, so this branch is a real work-in-progress and gets > force-pushed regularly :-) > > Roald > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Sep 9, 2013 at 1:16 AM, Sage Weil <sage@inktank.com> wrote: > I took a quick look and it looks pretty great. The builds are coming back > okay, too.. any compelling reason not to merge it now? Nope, not really. I'll do some last build checks at the end of today (monday over here) and then send you a pull request. > BTW, Roald, I think the obj_bencher cct change did something strange... > see http://tracker.ceph.com/issues/6256. Trying to reproduce it now. I was wondering when the first bug would show, but I see you've already fixed it... Roald -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Sep 9, 2013 at 1:27 AM, Milosz Tanski <milosz@adfin.com> wrote: > Are you using gold as a linker already instead of the traditional > binutils bdf based ld. It does make a difference esp on large C++ code > bases (where linking takes a lot of time). Actually I'm using by default on code bases the size of Ceph. For me, the time of a "make check" goes down from ~7m to ~6m, so that's a nice improvement. Might be a good idea to set up a gitbuilder instance using gold and let it running on the side for a while? Roald -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, Sep 7, 2013 at 10:39 PM, Sage Weil <sage@inktank.com> wrote: > I'm particularly keen on doing what we can to reduce build time. I also > agree there is a fair bit of tree restructuring that needs to be done. > (For example, libcommon.la is huge and includes lots of stuff that most > targets don't actually need; breaking this into smaller pieces would > probably help.) I've made some notes while cleaning up the automake stuff. It's on the wiki as a blueprint now so people can contribute and hopefully it will start a discussion about this; http://wiki.ceph.com/01Planning/02Blueprints/Emperor/Source_tree_restructuring Feel free to add/comment! Roald -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/src/Makefile.am b/src/Makefile.am index 93f3331..fb7c9dd 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -1,4 +1,4 @@ -AUTOMAKE_OPTIONS = gnu +AUTOMAKE_OPTIONS = gnu subdir-objects SUBDIRS = ocf java DIST_SUBDIRS = gtest ocf libs3 java