# 'MPI' has not been declared

31 posts / 0 new
'MPI' has not been declared
#1

Hello,

I have built the MPI binaries many times without issue... until now.  I have two machines with the same Ubuntu build, openmpi, and site.settings file on them.  The build goes fine on one, but fails on the other (I did the openmpi installation tests and they succeed).

In the build output I see: "'MPI' has not been declared".   Below is more of the output... thank you very much in advance for your time!

src/protocols/mpi_refinement/MPI_Refine_Master.cc: In member function 'virtual void protocols::mpi_refinement::MPI_Refine_Master::init()':
src/protocols/mpi_refinement/MPI_Refine_Master.cc:204:15: error: 'MPI' has not been declared
int ncores = MPI::COMM_WORLD.Get_size();
^

.....

scons: *** [build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/protocols/mpi_refinement/MPI_Refine_Master.os] Error 1
scons: building terminated because of errors."

Category:
Post Situation:
Wed, 2016-09-21 07:49
starone

What does "which mpicc" return?  What do the scons lines look like (the ones that probably start with mpicc)?  The individual scons lines should show paths to libraries - are the mpi libraries showing up like they should?  (You can determine "should" by comparison to the machine that works).

I assume the problem is that whatever the system is finding as MPI libraries aren't right, if they don't define MPI.  (I assume if it wasn't finding the libraries at all, it would complain that the library files weren't found).

It compiles fine without MPI, right?

Wed, 2016-09-21 07:57
smlewis

Thank you!

"which mpicc" returns: "/home/starone/openmpi/bin/mpicc"

Here is an individual line:

"mpiCC -o build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/protocols/mpi_refinement/MPI_Refine_Master.os -c -std=c++98 -ffor-scope -isystem external/boost_1_55_0/ -isystem external/ -isystem external/include/ -isystem external/dbio/ -pipe -Wall -Wextra -pedantic -Werror -Wno-long-long -Wno-strict-aliasing -march=core2 -mtune=generic -O3 -ffast-math -funroll-loops -finline-functions -finline-limit=20000 -s -Wno-unused-variable -Wno-unused-parameter -fPIC -DBOOST_ERROR_CODE_HEADER_ONLY -DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS -DPTR_BOOST -DNDEBUG -DUSEMPI -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc/5.4 -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -Iexternal/boost_1_55_0 -Iexternal/libxml2/include -Iexternal -Iexternal/dbio -I/usr/include -I/usr/local/include src/protocols/mpi_refinement/MPI_Refine_Master.cc"

Yes, it builds fine without MPI.

Wed, 2016-09-21 08:20
starone

Update, fixed:

I started looking through threads and found the following are needed... the machine I had the error on was a clean install of the OS so they were missing:

sudo apt-get install libopenmpi-dev

I also needed:

sudo apt-get install zlib1g-dev

Are these things always necessary?

Wed, 2016-09-21 10:46
starone

You will always need to install MPI to compile in MPI.  libopenmpi-dev is the stuff that gets #included, which is why you got those MPI not defiend errors.  I am updating the build documentation to clarify you do need the -dev package here.

You will always need zlib support for Rosetta.  zlib1g-dev is the usual ubuntu package that's necessary (and it's not usually present on ubuntu; it varies by system).  That's the only thing we consider an external dependency other than "a compiler".  (The feature this buys you is that Rosetta can basically always interpret gzipped inputs natively, and can optionally output many things as .gz as well).  Usually a missing zlib is exposed by "cannot find -lz", explained in the build documentation (https://www.rosettacommons.org/docs/latest/build_documentation/Build-Documentation#setting-up-rosetta-3_dependencies-troubleshooting)

Wed, 2016-09-21 11:27
smlewis

Great!  Thanks for the info and the update to the docs.

I actually thought I was in good shape because I saw no errors, but I also don't  see any .mpi executables... : (  I'll have to look into that tonight and tomorrow.

Wed, 2016-09-28 18:38
starone

Actually the standard binaries built (not sure what I messed up)... the MPI versions did not.  The error is the same "MPI has not been declared".

I thought maybe something went wrong with the libdev install so I ran it again but it appears to be fine unless I'm misinterpreting the following... so where else am I missing something?

"sudo apt-get install libopenmpi-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libopenmpi-dev is already the newest version (1.10.2-8ubuntu1).
The following packages were automatically installed and are no longer required:
libpango1.0-0 libpangox-1.0-0 linux-headers-4.4.0-21 linux-headers-4.4.0-21-generic linux-headers-4.4.0-24 linux-headers-4.4.0-24-generic
linux-image-4.4.0-21-generic linux-image-4.4.0-24-generic linux-image-extra-4.4.0-21-generic linux-image-extra-4.4.0-24-generic
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 29 not upgraded."

Thanks!

Wed, 2016-09-28 18:56
starone

On my VM here on my laptop, it builds fine… I noticed this difference between the build statements, but I don’t know why the path is missing from the build statement on my workstation... ($LD_LIBRARY_PATH and$PATH are set the same way on both machines)

Laptop:

“mpiCC -o build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/AnchoredDesign.mpi.linuxgccrelease -Wl,-rpath=/home/starone/Public/Rosetta/main/source/build/src/release/linux/4.4/64/x86/gcc/5.4/mpi -Wl,-rpath=/home/starone/Public/Rosetta/main/source/build/external/release/linux/4.4/64/x86/gcc/5.4/mpi -Wl,-rpath=\$ORIGIN -Wl,-rpath=\$ORIGIN/../lib build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/apps/public/interface_design/anchored_design/AnchoredDesign.o -Lexternal/lib -Lbuild/src/release/linux/4.4/64/x86/gcc/5.4/mpi -Lsrc -Lbuild/external/release/linux/4.4/64/x86/gcc/5.4/mpi -Lexternal -L/home/starone/openmpi/lib -L/home/starone/Public/PyRosetta.Namespace -L/home/starone/Public/PyRosetta.Namespace/rosetta -L/usr/lib -L/usr/local/lib -ldevel -lprotocols.7 -lprotocols.6 -lprotocols_e.5 -lprotocols_d.5 -lprotocols_c.5 -lprotocols_b.5 -lprotocols_a.5 -lprotocols_h.4 -lprotocols_g.4 -lprotocols_f.4 -lprotocols_e.4 -lprotocols_d.4 -lprotocols_c.4 -lprotocols_b.4 -lprotocols_a.4 -lprotocols.3 -lprotocols_b.2 -lprotocols_a.2 -lprotocols.1 -lcore.5 -lcore.4 -lcore.3 -lcore.2 -lcore.1 -lbasic -lnumeric -lutility -lObjexxFCL -lz -lcppdb -lsqlite3 -lcifparse….”

Workstation:  (-L/home/starone/openmpi/lib is missing)

“mpiCC -o build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/protocols/mpi_refinement/MPI_Refine_Master.os -c -std=c++98 -ffor-scope -isystem external/boost_1_55_0/ -isystem external/ -isystem external/include/ -isystem external/dbio/ -pipe -Wall -Wextra -pedantic -Werror -Wno-long-long -Wno-strict-aliasing -march=core2 -mtune=generic -O3 -ffast-math -funroll-loops -finline-functions -finline-limit=20000 -s -Wno-unused-variable -Wno-unused-parameter -fPIC -DBOOST_ERROR_CODE_HEADER_ONLY -DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS -DPTR_BOOST -DNDEBUG -DUSEMPI -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc/5.4 -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -Iexternal/boost_1_55_0 -Iexternal/libxml2/include -Iexternal -Iexternal/dbio -I/usr/include -I/usr/local/include src/protocols/mpi_refinement/MPI_Refine_Master.cc”

Thu, 2016-09-29 06:23
starone

The first thing to try when there are apparent pathing issues is to ln -s site.settings.killdevil site.settings.  This adds some of your enviroment variables into SCons in a different way.  (Those files are in source/tools/build).

Thu, 2016-09-29 07:24
smlewis

I tried the step you suggested (thank you!), but nothing changed.  One thing that is striking is the difference in the parameters... on my laptop I see "-L" parameters indicated which include all the elements of the LD LIBRARY PATH...  they aren't present on the workstation complie option list.

Why would this be?

Thu, 2016-09-29 10:12
starone

I'm surprised that the site.settings change did nothing - usually it forces a full recompile.  (I guess if it wasn't compiling before that's not noticeable).

Does the workstation have something unusual set up w/r/t the shell you are calling scons from versus the default shell for your user account?  Perhaps the environment variables don't exist in the scons environment if they aren't being set in your .zshrc or whatever?  (I'm guessing here.)

Thu, 2016-09-29 10:52
smlewis

I'm not a Linux person so I just accept whatever is installed when I installed the OS on the machines, the only changes I made were in the .bashrc files regarding the paths.  They are both Intel machines.

This is what I added to that file:  (sorry for the strange characters... hopefully you can figure it out.  I must be inadvertantly hitting on some escape characters or something)

source /home/starone/Public/PyRosetta.Namespace/SetPyRosettaEnvironment.sh

export PATH=/opt/ncbi-blast-2.3.0+/bin:$PATH export BLASTDB=/opt/ncbi-blast-2.3.0+/db export PATH=/opt/blast-2.2.26/bin:$PATH
export PATH=$HOME/openmpi/bin:$PATH
export LD_LIBRARY_PATH=$HOME/openmpi/lib:$LD_LIBRARY_PATH

These are the contents of the site.settings file I have been using on both machines:

import os
settings = {
"site" : {
"prepends" : {
"program_path" : os.environ["PATH"].split(":"),
"library_path" : os.environ["LD_LIBRARY_PATH"].split(":"),
},
"appends" : {
},
"overrides" : {
},
"removes" : {
},
},
}

Thu, 2016-09-29 11:05
starone

The SetPyRosettaEnvironment.sh shouldn't do anything with respect to an MPI compile - that just sets things up for PyRosetta, and the compilation of commandline Rosetta is completely separate from it.

Just to confirm, the error message you're getting is still the same as the one at the top of this thread, right? The "error: 'MPI' has not been declared" in MPI_Refine_Master.cc (or similar), right?

If that's the case, that's a header issue, and not a library issue -- the -L settings and the LD_LIBRARY_PATH shouldn't affect that. I'd focus more on the "-I" settings for the include path. (One complication here is that you showed a linker commandline for the laptop compile, but a compile commandline for the workstation - that's why you see the -L's in one but not in the other.)

My initial guess would be that the path to the MPI includes is missing. That'd be a little odd, though, as you would expect that you'd get a different error in that case. But it might be an issue of *where* the MPI libraries you're using are located.

If you're using the Ubuntu system libopenmpi, then it looks like the headers should be (for Xenial) in /usr/lib/openmpi/include/ It doesn't look like you have that enabled. I'm not sure where it's pulling  the mpi.hh from, though: probably not from where you expect it. Try adding

"include_path" : ["/usr/lib/openmpi/include/", ],

To the "prepends" section of your site.settings file and recompiling.

I'm not sure why it's working on your laptop, though, if you're using the same packages. Perhaps because you did a manual install of the library in /home/starone/openmpi/ and it's pulling from that. (If you do want to use the manually installed version rather than the system version, you'll need to change the paths.)

Thu, 2016-09-29 12:09
rmoretti

yes, same error

Thu, 2016-09-29 18:56
starone

Ok... I think I see what you are getting at.  I will try it in the morning.

(sorry if there is and I just don't know about it):  I haven't been an engineer for many years, but when I have to build something in Linux I always wonder why there isn't something slick like MS's Visual Studio.  I did Microsoft development for a long time before moving into management and the ability to visualize things from within a development studio is so much more helpful to me than using command line interactions... feel like I'm back in the MS-DOS era : )

Thanks again for your help!

Thu, 2016-09-29 18:56
starone

Progress!  Well, the compilation looks like its succeeding now, but the linking is failing... pretty much as expected I suppose.  I think you're right that I used different methods for installing openmpi (I installed on the laptop a while ago), but it wasn't by choice.  There is a good deal of conflicting info on the web on this stuff as I'm sure you know.  On the workstation I downloaded the package from the OpenMPI website and followed the steps in the YouTube video linked there.

I noticed that there are about 80 items in openmpi/lib directory on my laptop, but only about 20 on the workstation.

I documented the steps I followed... appearently they are incorrect, can you tell me the proper method.

1. Download the latest stable build tar.gz

2. Make directory Public/openmpi

3. Copy the compressed file to the directory

4. Open a terminal window

5. Change to the openmpi directory

6. Execute: ./configure --prefix=$HOME/openmpi 7. Execute: make all 8. Execute: make install 9. Change to your home directory 10. Execute: sudo gedit .bashrc 11. Add the following lines to the end of the file: export PATH=$HOME/openmpi/bin:$PATH export LD_LIBRARY_PATH=$HOME/openmpi/lib:$LD_LIBRARY_PATH 12. Save and close the editor 13. Execute: mpirun -np n hostname (where 'n' is the number of processors in your machine) These are the errors I'm getting now: mpiCC -o build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/per_residue_energies.mpi.linuxgccrelease -Wl,-rpath=/home/starone/Public/Rosetta/main/source/build/src/release/linux/4.4/64/x86/gcc/5.4/mpi -Wl,-rpath=/home/starone/Public/Rosetta/main/source/build/external/release/linux/4.4/64/x86/gcc/5.4/mpi -Wl,-rpath=\$ORIGIN -Wl,-rpath=\$ORIGIN/../lib build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/apps/public/analysis/per_residue_energies.o -Lexternal/lib -Lbuild/src/release/linux/4.4/64/x86/gcc/5.4/mpi -Lsrc -Lbuild/external/release/linux/4.4/64/x86/gcc/5.4/mpi -Lexternal -L/home/starone/Public/PyRosetta.Namespace -L/home/starone/Public/PyRosetta.Namespace/rosetta -L/home/starone/openmpi/lib -L/usr/lib -L/usr/local/lib -ldevel -lprotocols.7 -lprotocols.6 -lprotocols_e.5 -lprotocols_d.5 -lprotocols_c.5 -lprotocols_b.5 -lprotocols_a.5 -lprotocols_h.4 -lprotocols_g.4 -lprotocols_f.4 -lprotocols_e.4 -lprotocols_d.4 -lprotocols_c.4 -lprotocols_b.4 -lprotocols_a.4 -lprotocols.3 -lprotocols_b.2 -lprotocols_a.2 -lprotocols.1 -lcore.5 -lcore.4 -lcore.3 -lcore.2 -lcore.1 -lbasic -lnumeric -lutility -lObjexxFCL -lz -lcppdb -lsqlite3 -lcifparse -lxml2 build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.3.so: undefined reference to ompi_mpi_cxx_op_intercept' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.3.so: undefined reference to MPI::Win::Free()' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.3.so: undefined reference to MPI::Datatype::Free()' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.6.so: undefined reference to MPI::COMM_WORLD' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.3.so: undefined reference to MPI::Comm::Comm()' collect2: error: ld returned 1 exit status scons: *** [build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/helix_from_sequence.mpi.linuxgccrelease] Error 1 build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/apps/public/design/pmut_scan_parallel.o: In function MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype const&), bool)': pmut_scan_parallel.cc:(.text._ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb[_ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb]+0x17): undefined reference to ompi_mpi_cxx_op_intercept' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/apps/public/design/pmut_scan_parallel.o: In function MPI::Intracomm::Clone() const': pmut_scan_parallel.cc:(.text._ZNK3MPI9Intracomm5CloneEv[_ZNK3MPI9Intracomm5CloneEv]+0x3a): undefined reference to MPI::Comm::Comm()' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/apps/public/design/pmut_scan_parallel.o: In function MPI::Graphcomm::Clone() const': pmut_scan_parallel.cc:(.text._ZNK3MPI9Graphcomm5CloneEv[_ZNK3MPI9Graphcomm5CloneEv]+0x35): undefined reference to MPI::Comm::Comm()' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/apps/public/design/pmut_scan_parallel.o: In function MPI::Cartcomm::Sub(bool const*) const': pmut_scan_parallel.cc:(.text._ZNK3MPI8Cartcomm3SubEPKb[_ZNK3MPI8Cartcomm3SubEPKb]+0x194): undefined reference to MPI::Comm::Comm()' Fri, 2016-09-30 05:58 starone Wait, why are we talking about manually installed openmpi? I thought you installed it via apt? "sudo apt-get install openmpi openmpi-dev" or something similar? Fri, 2016-09-30 07:39 smlewis Not on the workstation... I followed those steps because when I tried sudo apt-get install openmpi I think I remember it saying it wasn't found. I found a page today that indicates that I have to use 'sudo apt-get install openmpi-bin'... is that right? Fri, 2016-09-30 07:46 starone I can't tell you what's right; I haven't installed MPI on linux in years. What you just said is similar to what I suggested so it's worth trying. You'll want the -dev package too, remember. You should be able to do something like "apt list | grep mpi" on the machine that works to see which mpi packages you installed, and just do the same on the other machine...? Fri, 2016-09-30 08:59 smlewis Thanks! I'll give it a try Fri, 2016-09-30 09:35 starone I'd definitely recommend using the system versions (the one from apt-get) if you can. MPI is a little bit finicky at times, and it's a lot easier if you leave setting things up properly to people who know the operating system well, rather than trying to get things set up appropriately yourself. In fact, if you continue to have issues, I'd recommend removing/uninstalling the manually installed version, to make sure that having two versions of the MPI libraries aren't confusing things. (You can get issues if Rosetta is picking up the headers of one and the libraries from the other.) Fri, 2016-09-30 09:56 rmoretti I tried using sudo apt-get... it just reports that the packages are already installed. These errors are different than I was getting before... now I'm seeing alot of this: scons: *** [build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/beta_peptide_modeling.mpi.linuxgccrelease] Error 1 build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libutility.so: undefined reference to ompi_mpi_cxx_op_intercept' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libutility.so: undefined reference to MPI::Datatype::Free()' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libutility.so: undefined reference to MPI::Comm::Comm()' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libutility.so: undefined reference to MPI::Win::Free()' collect2: error: ld returned 1 exit status scons: *** [build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/antibody.mpi.linuxgccrelease] Error 1 build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.1.so: undefined reference to ompi_mpi_cxx_op_intercept' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.1.so: undefined reference to MPI::Win::Free()' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.1.so: undefined reference to MPI::Datatype::Free()' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.6.so: undefined reference to MPI::COMM_WORLD' build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.1.so: undefined reference to MPI::Comm::Comm()' collect2: error: ld returned 1 exit status Fri, 2016-09-30 10:09 starone Okay, *now* you're getting issues with your -L settings. The killdevil settings should have fixed this, but you might have to supplement this with the openmpi location "library_path" : os.environ["LD_LIBRARY_PATH"].split(":") + ["/usr/lib/openmpi/lib/",], If that works for compilation, in order to run things successfully, you may need to actually add /usr/lib/openmpi/lib/ to your LD_LIBRARY path export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/openmpi/lib/

You can do this either any time you need to run MPI, or you could add it to your .bashrc to have it on permanently.

Sat, 2016-10-01 11:26
rmoretti

Thank you!  I will give that a try in the morning.

One odd thing I noticed is that on my laptop there are about 79 items in the openmpi bin folder but on the workstation there are less than half that number.  I looked at the build lines and openmpi is one of the -L options.

I will try what you suggested but I was hoping for your thoughts on the differening file counts... maybe the path is correct but the libraries are missing somehow?

Sun, 2016-10-02 18:36
starone

Are these both apt-get installed packages? Or are you comparing a manual install to the apt-get install?

I would not be surprised if the apt-get install is smaller (or larger). The people who do the Ubuntu packaging probably know which files are required and which are extra, and remove anything that is not really necessary for general Ubuntu usage. (Or alternatively, there may be multiple ways to do the build, and the Ubuntu packagers do all of them, as they don't know which ones would really be needed for your particular use case.)

Mon, 2016-10-03 10:54
rmoretti

On the workstation (problem machine) I first did a manual install using the steps listed in #16 above that I found in a video linked from the openmpi website.  The generated fewer files.  On my laptop I did the apt-get method there are more files and everything works fine.

Sorry I didn't get to try your suggestion as I woke up this morning feeling horrible and went back to bed after the kids were off to school.  Hopefully I'll be feeling better tomorrow so I can have at it.

Mon, 2016-10-03 19:05
starone

Thanks, but no changes...

Ok, as I see it I have one machine where the build process can find, for example, "MPI::COMM_WORLD" and another on which it can't.  Both machines have the same OpenMPI package installed and both have the same library paths defined.

So, what I'd like to do is find out from which library file, on the machine where the build works, exports "MPI::COMM_WORLD".  This way I can see if it exists on the machine where it wont build.

Tue, 2016-10-04 07:21
starone

Maybe this explains something?

Both machines are running Ubuntu 16.04, but my laptop (where everthing builds fine) was an upgrade, while I did a clean install on my workstation (where the build fails).  Everything else between the machines in terms of software, shells, settings, etc, are the same.

Wed, 2016-10-05 06:15
starone

Does anyone know about installing OpenMPI?  I ran 'ompi_info' on both machines and the installation options and version info are different.  I attached both files for review.

CatalayseU:  Laptop, build successful

SynthaseU: Workstation, build fails

Thank you!!

File attachments:
Wed, 2016-10-05 06:43
starone

Could this "C++ bindings: no" be the problem?

If it is, does anyone know how to change it and rebuild?

Thanks

Wed, 2016-10-05 06:59
starone

Ok, looks like this is what happened...  The C++ bindings are off by default when building OpenMPI (though I can't imagine why!).  So I redid the build with this added to the ./configure command:  "--enable-mpi-cxx".

Then I redid the remaining steps as I outlined above.  I then did the MPI build of Rosetta and it was successful.

How do I mark this as answered?

Thanks for your help guys.

Wed, 2016-10-05 07:26
starone