Code Monkey home page Code Monkey logo

Comments (3)

AndreaValenti01 avatar AndreaValenti01 commented on June 2, 2024

I've the same issue.
@dedalo944 have you found a solution? The latest version works well on Windows but not on linux distro (i use debian)

from core.

dedalo944 avatar dedalo944 commented on June 2, 2024

I've the same issue. @dedalo944 have you found a solution? The latest version works well on Windows but not on linux distro (i use debian)
this one into upload.py, for now

def deep_copy_upload_file(upload_file: UploadFile) -> UploadFile:
    file_bytes = upload_file.file.read()
    
    new_upload_file = UploadFile(
        filename=upload_file.filename,
        file=io.BytesIO(file_bytes)
    )
    
    return new_upload_file

# receive files via http endpoint
@router.post("/")
async def upload_file(
    request: Request,
    file: UploadFile,
    background_tasks: BackgroundTasks,
    chunk_size: int | None = Body(
        default=None,
        description="Maximum length of each chunk after the document is split (in tokens)",
    ),
    chunk_overlap: int | None = Body(default=None, description="Chunk overlap (in tokens)"),
    stray = Depends(session),
) -> Dict:
    """Upload a file containing text (.txt, .md, .pdf, etc.). File content will be extracted and segmented into chunks.
    Chunks will be then vectorized and stored into documents memory.
    """

    # Check the file format is supported
    admitted_types = stray.rabbit_hole.file_handlers.keys()

    # Get file mime type
    content_type = mimetypes.guess_type(file.filename)[0]
    log.info(f"Uploaded {content_type} down the rabbit hole")

    # check if MIME type of uploaded file is supported
    if content_type not in admitted_types:
        raise HTTPException(
            status_code=400,
            detail={
                "error": f'MIME type {content_type} not supported. Admitted types: {" - ".join(admitted_types)}'}
        )

    # upload file to long term memory, in the background
    background_tasks.add_task(
        # we deepcopy the file because FastAPI does not keep the file in memory after the response returns to the client
        # https://github.com/tiangolo/fastapi/discussions/10936
        stray.rabbit_hole.ingest_file, stray, deep_copy_upload_file(file), chunk_size, chunk_overlap
    )

    # reply to client
    return {
        "filename": file.filename,
        "content_type": file.content_type,
        "info": "File is being ingested asynchronously",
    }

from core.

pieroit avatar pieroit commented on June 2, 2024

Fixed in develop, coming soon to next release

from core.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.