ERROR: invalid function number 4, must be between 1 and 3 about lsm3 HOT 5 OPEN

Sorry, I could not reproduce the problem.
I have tested LSM3 both with current master and REL_13_STABLE branch.
How are you building lsm3 extension?

from lsm3.

After cloning repository i follow these steps in postgresql 13.0 ubuntu.

1. cd lsm3
2. make
3. make install
4. copied lines from lsm3.conf and added them to install/data/postgres.conf
5. restart postgres server
   6.tried both make installcheck and create extension lsm3 in psql both are failing .

Please check and tell me that i am following the required steps

from lsm3.

Do you have Postgres development package (with sources).
To build extension you need to have Postgres sources.
It is not clear from your explanation above whether lsm3 was build in 13.0 sources tree.
Lsm3 extension can be built using PGXS (make USE_PGXS=1) or should be placed in contrib directory of Postgres source tree. Did you use second approach? If so where did you get postgres sources? From main Postgres repository
git://git.postgresql.org/git/postgresql.git ?
If so, can you perform "git status" in it?

from lsm3.

i use first approch. let me rephrase

1. use make USE_PGXS=1
2. make install USE_PGXS=1
3. make installcheck USE_PGXS=1

i am able to create other extension like base36

from lsm3.

In case of using PGXS you need to ensure that path is pointed to correct postgres version.
Can you please do "type postgres" and then "postgres --version" ?

from lsm3.