scaledfoundations / grid-playground Goto Github PK

is It possible to use other LLM in the simulator, For example, in scaledfoundations grid:base. Could I modify the config/airgen_config.yml or somewhere to use other LLM.

What is the licensing terms for Airgen for students? Thinking on implementing land vehicles.

Hey,

I'm diving into your project, specifically looking at the different .py modules. Ran into a snag in the Docker setup – it's all .pyc files, no .py ones. This makes it tough to get the full picture and think about contributing.

Any chance you'll be releasing the .py source files? Or can you point me in the right direction to find them? I'm mainly looking at this for learning and maybe pitching in down the line.

Thanks!

Traceback (most recent call last): File "/GRID/grid_chat.py", line 6, in <module> run_chat(config_path) File "build/bdist.linux-x86_64/egg/grid/chat.py", line 113, in run_chat File "build/bdist.linux-x86_64/egg/grid/orchestrator.py", line 171, in __enter__ AssertionError: Error in the initialization code block: An error occurred while executing the code: Traceback (most recent call last) Cell In[3], line 4 2 from grid.model.model import ModelCollection 3 import math, numpy ----> 4 airgendrone = AirGenDrone() 5 modelcollection=ModelCollection(["seg-gsam","vqa-git","nav-inspect","detection-gdino"]) File build/bdist.linux-x86_64/egg/grid/robot/airgen_drone.py:46, in __init__(self, visual_assistant) File /opt/conda/lib/python3.11/site-packages/airgen/client.py:2696, in connect_airgen(robot_type, ip_address, port, geo, run_in_background) 2694 else: 2695 client = robot_client_class(ip=host, port=port) -> 2696 client.confirmConnection() 2697 ## temp fix that the first segmentation image is not properly rendered 2698 client.simGetImages( 2699 [ImageRequest("front_center", ImageType.Segmentation, False, False)] 2700 ) File /opt/conda/lib/python3.11/site-packages/airgen/client.py:165, in VehicleClient.confirmConnection(self) 163 info: List[str] = [] 164 err_msg: List[str] = [] --> 165 if self.ping(): 166 info.append("Connected!") 167 else: File /opt/conda/lib/python3.11/site-packages/airgen/client.py:59, in VehicleClient.ping(self) 52 def ping(self): 53 """ 54 If connection is established then this call will return true otherwise it will be blocked until timeout 55 56 Returns: 57 bool: 58 """ ---> 59 return self.client.call("ping") File /opt/conda/lib/python3.11/site-packages/airgen_third_party/msgpackrpc/session.py:42, in Session.call(self, method, *args) 41 def call(self, method, *args): ---> 42 return self.send_request(method, args).get() File /opt/conda/lib/python3.11/site-packages/airgen_third_party/msgpackrpc/future.py:55, in Future.get(self) 53 else: 54 if isinstance(self._error, error.RPCError): ---> 55 raise self._error 56 else: 57 raise error.RPCError(self._error) TransportError: Retry connection over the limit

I start the Neighborhood environment in a terminal with bash run.sh. In a different terminal I run python 04_path_planning.py. This results in the below error-

Failed to find symbol file, expected location:
"/home/local/xxx/Simulators/AirGen/ModularNeighborhoodPack_Linux/AirGenEnv/Binaries/Linux/AirGenEnv.sym"
Assertion failed: !IsGarbageCollectingAndLockingUObjectHashTables() [File:./Runtime/CoreUObject/Private/UObject/UObjectGlobals.cpp] [Line: 3238]
Unable to create new object: CatmullRomSpline /Engine/Transient.None. Creating UObjects while Collecting Garbage is not allowed!
[2023.12.15-22.34.01:357][618]LogStreaming: Display: 0.007 ms for processing 136 objects in RemoveUnreachableObjects(Queued=0, Async=0). Removed 3 (1277->1274) packages and 121 (2365->2244) public exports.
[2023.12.15-22.34.01:357][618]LogCore: Error: appError called: Assertion failed: !IsGarbageCollectingAndLockingUObjectHashTables() [File:./Runtime/CoreUObject/Private/UObject/UObjectGlobals.cpp] [Line: 3238]
Unable to create new object: CatmullRomSpline /Engine/Transient.None. Creating UObjects while Collecting Garbage is not allowed!

Signal 11 caught.
Malloc Size=262146 LargeMemoryPoolOffset=262162
CommonUnixCrashHandler: Signal=11
[2023.12.15-22.34.01:367][618]LogCore: === Critical error: ===
Unhandled Exception: SIGSEGV: invalid attempt to write memory at address 0x0000000000000003

[2023.12.15-22.34.01:367][618]LogCore: Assertion failed: !IsGarbageCollectingAndLockingUObjectHashTables() [File:./Runtime/CoreUObject/Private/UObject/UObjectGlobals.cpp] [Line: 3238]
Unable to create new object: CatmullRomSpline /Engine/Transient.None. Creating UObjects while Collecting Garbage is not allowed!
0x0000000007411ecd AirGenEnv!UnknownFunction(0x7211ecc)
0x000000000740f2d5 AirGenEnv!UnknownFunction(0x720f2d4)
0x0000000004814db7 AirGenEnv!UnknownFunction(0x4614db6)
0x00000000047f96a4 AirGenEnv!UnknownFunction(0x45f96a3)
0x00000000046e655a AirGenEnv!UnknownFunction(0x44e6559)
0x00000000046e6c0c AirGenEnv!UnknownFunction(0x44e6c0b)
0x000000000474e946 AirGenEnv!UnknownFunction(0x454e945)
0x000000000cd3d759 AirGenEnv!UnknownFunction(0xcb3d758)
0x000000000cd7ea34 AirGenEnv!_ZN3rpc6detail10dispatcher13dispatch_callERKN14clmdep_msgpack2v26objectEb(+0x83)
0x000000000cd7e79c AirGenEnv!_ZN3rpc6detail10dispatcher8dispatchERKN14clmdep_msgpack2v26objectEb(+0x2b)
0x000000000cd8d094 AirGenEnv!UnknownFunction(0xcb8d093)
0x000000000cd89819 AirGenEnv!_ZN11clmdep_asio6detail15task_io_service10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_27task_io_service_thread_infoERKNSt3__110error_codeE(+0x358)
0x000000000cd89331 AirGenEnv!_ZN11clmdep_asio6detail15task_io_service3runERNSt3__110error_codeE(+0xa0)
0x000000000cd8346e AirGenEnv!UnknownFunction(0xcb8346d)
0x000000000cd8a1f0 AirGenEnv!UnknownFunction(0xcb8a1ef)
0x00007f8e13e94ac3 libc.so.6!UnknownFunction(0x94ac2)
0x00007f8e13f26660 libc.so.6!UnknownFunction(0x12665f)

0x0000000007411ecd AirGenEnv!UnknownFunction(0x7211ecc)
0x000000000740f2d5 AirGenEnv!UnknownFunction(0x720f2d4)
0x0000000004814db7 AirGenEnv!UnknownFunction(0x4614db6)
0x00000000047f96a4 AirGenEnv!UnknownFunction(0x45f96a3)
0x00000000046e655a AirGenEnv!UnknownFunction(0x44e6559)
0x00000000046e6c0c AirGenEnv!UnknownFunction(0x44e6c0b)
0x000000000474e946 AirGenEnv!UnknownFunction(0x454e945)
0x000000000cd3d759 AirGenEnv!UnknownFunction(0xcb3d758)
0x000000000cd7ea34 AirGenEnv!_ZN3rpc6detail10dispatcher13dispatch_callERKN14clmdep_msgpack2v26objectEb(+0x83)
0x000000000cd7e79c AirGenEnv!_ZN3rpc6detail10dispatcher8dispatchERKN14clmdep_msgpack2v26objectEb(+0x2b)
0x000000000cd8d094 AirGenEnv!UnknownFunction(0xcb8d093)
0x000000000cd89819 AirGenEnv!_ZN11clmdep_asio6detail15task_io_service10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_27task_io_service_thread_infoERKNSt3__110error_codeE(+0x358)
0x000000000cd89331 AirGenEnv!_ZN11clmdep_asio6detail15task_io_service3runERNSt3__110error_codeE(+0xa0)
0x000000000cd8346e AirGenEnv!UnknownFunction(0xcb8346d)
0x000000000cd8a1f0 AirGenEnv!UnknownFunction(0xcb8a1ef)
0x00007f8e13e94ac3 libc.so.6!UnknownFunction(0x94ac2)
0x00007f8e13f26660 libc.so.6!UnknownFunction(0x12665f)

[2023.12.15-22.34.01:378][618]LogExit: Executing StaticShutdownAfterError
Malloc Size=131160 LargeMemoryPoolOffset=393352
Malloc Size=131160 LargeMemoryPoolOffset=524536
Malloc Size=1048576 LargeMemoryPoolOffset=1573136
[2023.12.15-22.34.01:392][618]LogHAL: Error: FUnixPlatformProcess::CreateProc: File does not exist (/home/local/xxx/Simulators/AirGen/ModularNeighborhoodPack_Linux/Engine/Binaries/Linux/CrashReportClient)
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault (core dumped)

System specs:

Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

Please let me know if I'm doing something foolish. Thanks for your help!

On the first step:

docker pull scaledfoundations/grid:base

docker run -it --gpus all --network host grid:base

The issue here is the way you guys suggested to be referencing the Docker image. The image is pulled from scaledfoundations/grid:base, but when you run the docker run command, you're only specifying grid:base, which Docker can't find locally or in the default Docker registry.

You should use the full name of the image including the repository when running it:

docker run -it --gpus all --network host scaledfoundations/grid:base

This will tell Docker to use the scaledfoundations/grid:base image that you've pulled, and it should be able to locate it and run it successfully.

When trying to run the command: "docker run -it --gpus all --network host scaledfoundations/grid:base" on windows.

I'm getting: "docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/31d71227ce171f1818d76d742059bb590efbdb1da6958bc1ca5fc055a79ef6e6/merged/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1: file exists: unknown."

AirGenDemo works fine though.

[grid][2023-10-23 07:49:52,675][INFO]-[ init.py: 75] - disabled transformers's warning
[grid][2023-10-23 07:49:58,080][INFO]-[ orchestrator.py: 140] - Robot ready
[grid][2023-10-23 07:49:58,081][INFO]-[ orchestrator.py: 144] - World ready
[2023-10-23 07:49:59,728] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[grid][2023-10-23 07:50:03,123][WARNING]-[objectinspection.py: 32] - torch or pytorch3d is not installed: No module named 'pytorch3d'
final text_encoder_type: bert-base-uncased
[grid][2023-10-23 07:50:27,031][INFO]-[ sys_utils.py: 74] - method: init of class: ObjectInspect has no docstring
[grid][2023-10-23 07:50:27,031][INFO]-[ orchestrator.py: 156] - Models ready
GRID> what do you see
Thinking...