The following expression is possible for MySQL for example:

    CREATE TABLE IF NOT EXISTS table_name(...);

sqlparse does not recognize any statements after such expression. Some code snippet to reproduce:

    import sqlparse
    a = sqlparse.split("""
        drop table a; 
        create table if not exists a ( id int );
        dropt table b;""")
    len(a) # len(a) = 2, should be 3 here

Although the support of SQL standards in MySQL is well known (it's bad) such constructions can appear in database dump for sure.

<b>What steps will reproduce the problem?</b>
python -c 'import sqlparse; print sqlparse.format(&quot;SELECT foo, null bar, car FROM dual&quot;, reindent = True);'

<b>What is the expected output? What do you see instead?</b>
I would expect something like:

SELECT      foo,
       null bar,
            car
FROM dual

but instead get:

SELECT foo,
       null bar,
            car
FROM dual

<b>What version of the product are you using? On what operating system?</b>
Tested with both 0.1.3 and head on CentOS 5.4, python 2.7.

<b>What steps will reproduce the problem?</b>
>>> import sqlparse
>>> print(sqlparse.format('select -- a comment\nfrom foo', reindent=True))
select -- a comment

from foo
>>> 

<b>What is the expected output? What do you see instead?</b>
There should be no additional new line after &quot;-- a comment&quot;.


<b>Please use labels and text to provide additional information.</b>

Original query: select m.id, m.name from main m where m.open=1

Invocation arguments: sqlformat -k upper -i lower -r --indent_width=4 -

Resulting query:

SELECT M.id, M.name
FROM main M
WHERE M.OPEN=1

Expected result:

SELECT m.id, m.name
FROM main m
WHERE m.open=1

<b>What steps will reproduce the problem?</b>
1. echo 'select &quot;foo&quot; from bar'|bin/sqlformat -i &quot;upper&quot; -
<b>2.</b>
<b>3.</b>

<b>What is the expected output? What do you see instead?</b>

ouput: select &quot;FOO&quot; from BAR

&quot;foo&quot; shouldn't be changed at all.


<b>Please use labels and text to provide additional information.</b>

When parsing a query like this:

CREATE TABLE example_table AS SELECT * from example_table_2;

This token list is produced:

[<DDL 'CREATE' at 0x297c690>,
 <Whitespace ' ' at 0x297c650>,
 <Keyword 'TABLE' at 0x297c750>,
 <Whitespace ' ' at 0x297c790>,
 <Identifier 'exampl...' at 0x2b9a7c0>,
 <Whitespace ' ' at 0x297c910>,
 <Wildcard '*' at 0x297c950>,
 <Whitespace ' ' at 0x297c990>,
 <Keyword 'from' at 0x297c9d0>,
 <Whitespace ' ' at 0x297ca10>,
 <Identifier 'exampl...' at 0x2b9a830>,
 <Punctuation ';' at 0x297cad0>]

Notice that in the above list:
 <Identifier 'exampl...' at 0x2b9a7c0>,

has &quot;consumed&quot; the select DDL. Below are the tokens in <Identifier 'exampl... :

[<Name 'exampl...' at 0x297c7d0>,
 <Whitespace ' ' at 0x297c810>,
 <Keyword 'AS' at 0x297c850>,
 <Whitespace ' ' at 0x297c890>,
 <DML 'SELECT' at 0x297c8d0>]

The solution is for the identifiers to get grouped only when a non sql
reserved keyword follows.

<b>What steps will reproduce the problem?</b>
1. parse a CREATE TABLE statement with a VARCHAR2
<b>2.</b>
<b>3.</b>

<b>What is the expected output? What do you see instead?</b>
VARCHAR2 is not recognized as Token.Name.Builtin

<b>What version of the product are you using? On what operating system?</b>
0.1.1

<b>Please provide any additional information below.</b>

At this moment, a filter is a class with just a process() method inside it. I propose to use instead callable objects (objects with a __call__() method) so it's more standard and the most simple filters would be regular python functions since in fact functions in python are callable objects.

Also, allowing the pipeline system that the stream is mutable it would allow to use directly functions of other libraries without glue code. The only thing is that the functions should return a generator for optimization, but this is not neccesary.

As a last point, it should be set as deprecated the process() style. At this moment the pipeline system is backbard compatible and it wouldn't be difficult make it also the full library, but in a later version i think it should be necesary.

<b>What steps will reproduce the problem?</b>
1. Create sql file with comments in russian encoded with cp1251
2. Feed it to formatter

<b>What is the expected output? What do you see instead?</b>
I expect comments to stay &quot;as is&quot;. I see python stack trace instead.

<b>What version of the product are you using? On what operating system?</b>
Current version; win32 platform.

Example statement: "UPDATE foo SET foo_id=foo_id+3000;"

The Statement class should provide functions to programmatically retrieve
column definitions.

The attached file contains an example implementation of the low-level
parsing. See this thread on the mailing list for API ideas: 
http://groups.google.com/group/sqlparse/browse_thread/thread/4c1e6e22484ad07c

some variants of SQL allow [name with blanks] to specify a name with embedded blanks.

you could say:
str = '''select a.[fist name], a.[last name], a.city
from tab;'''


sqlparse.format(str, reindent=True, indent_tabs=False)
returns:
select a.[fist name] as field1,
                   a.[last name] as field2,
                           a.city
from tab;

Sqlparse puts the right parenthesis inside a 'Where' group for certain queries

<b>What steps will reproduce the problem?</b>

> s=sqlparse.parse('select * from (select * from test where rowid!=1)')[0]
> s.tokens[6]
<Parenthesis '(selec...' at 0x1df8050>
> s.tokens[6].tokens 
[<Punctuation '(' at 0x1dfba70>,
 <DML 'select' at 0x1dfb998>,
 <Whitespace ' ' at 0x1dfb908>,
 <Wildcard '*' at 0x1dfb9e0>,
 <Whitespace ' ' at 0x1dfb950>,
 <Keyword 'from' at 0x1dfbab8>,
 <Whitespace ' ' at 0x1dfbb00>,
 <Identifier 'test' at 0x1df8320>,
 <Whitespace ' ' at 0x1dfbb90>,
 <Where 'where ...' at 0x1df8140>]

<b>What is the expected output? What do you see instead?</b>
I would expect at the end of the parenthesis group (above) to see something
like:
<Punctuation ')' at 0xxxxxxx>

<b>What version of the product are you using? On what operating system?</b>
2 months old sqlparse from the hg repository on linux

<b>Please provide any additional information below.</b>

<b>What steps will reproduce the problem?</b>
<b>1.</b>
<b>2.</b>
<b>3.</b>

<b>What is the expected output? What do you see instead?</b>


<b>What version of the product are you using? On what operating system?</b>


<b>Please provide any additional information below.</b>

<b>What steps will reproduce the problem?</b>
1. Try to split a query with several statements using a StatementFilter() (in my case, i use it indirectly throught split2() ) when there are no space between the last word of the statement and the semicolon.

<b>What is the expected output? What do you see instead?</b>
It should split the statements correctly, but in fact, it thread the word and the semicolon as an only one token. If you put a space between them, it split correctly showing me two statements inside the query.

<b>What version of the product are you using? On what operating system?</b>
SVN HEAD version, Ubuntu 11.04 x64

<b>Please provide any additional information below.</b>

This is the working query with the space after the INSERT query, if you remove it it doesn't work:

-- Split the chunks in the database in two (old-head and new-tail)
-- based on it's defined length

-- Create new chunks containing the tail sectors and
-- update the old chunks length to contain only the head sectors

INSERT INTO chunks(file, block,           length,           sector)
            SELECT file, block+:length+1, length-:length-1, sector+:length+1
            FROM chunks
            WHERE file IS :file
              AND block = :block ;

UPDATE chunks SET length  = :length
WHERE file IS :file
  AND block = :block;

<b>What steps will reproduce the problem?</b>
1. parse a CREATE TABLE statement with a VARCHAR2
<b>2.</b>
<b>3.</b>

<b>What is the expected output? What do you see instead?</b>
VARCHAR2 is not recognized as Token.Name.Builtin

<b>What version of the product are you using? On what operating system?</b>
0.1.1

<b>Please provide any additional information below.</b>

>>> sqlparse.parse('--hello')[0].tokens[0].tokens
[<Operator '-' at 0x100550ed0>, <Operator '-' at 0x100550f10>, <Identifier 'hello' at 0x100548c20>]
>>> sqlparse.parse('--hello\n\n')[0].tokens
[<Single '--hell...' at 0x100562090>, <Newline ' ' at 0x100562190>]
>>> sqlparse.parse('--')[0].tokens
[<Operator '-' at 0x1005621d0>, <Operator '-' at 0x100562210>]

however, add even an invalid statement and suddenly the comment can be parsed:

>>> sqlparse.parse('--\n;')[0].tokens
[<Comment '-- ' at 0x100548c20>, <Punctuation ';' at 0x100562250>]

<b>What steps will reproduce the problem?</b>

python -c 'import sqlparse; print sqlparse.format (&quot;SELECT t.c1, t.c2, COALESCE(t.c3, t.c4), COALESCE(t.c5, t.c6), t.c7 FROM t;&quot;, reindent = True);'


<b>What is the expected output? What do you see instead?</b>

The expected output would be left-aligning &quot;t.&quot; and COALESCE:

| SELECT t.c1,
|        t.c2,
|        COALESCE(t.c3, t.c4),
|        COALESCE(t.c5, t.c6),
|        t.c7
| FROM t;

Instead I see a &quot;stair&quot;:

| SELECT t.c1,
|          t.c2,
|            COALESCE(t.c3, t.c4),
|            COALESCE(t.c5, t.c6),
|            t.c7
| FROM t;


<b>What version of the product are you using? On what operating system?</b>

0.1.3 on Python 2.6 (Linux).


<b>Please provide any additional information below.</b>

The formatting of &quot;SELECT c1, c2, COALESCE(c3, c4), COALESCE(c5, c6), c7 FROM t;&quot; and &quot;SELECT t.c1, t.c2, t.c3, t.c5, t.c7 FROM t;&quot; is &quot;correct&quot; (= to my taste :-)).

<b>What steps will reproduce the problem?</b>
Run the command:
python -c 'import sqlparse; print sqlparse.format(&quot;SELECT foo, bar, CASE WHEN foo < bar THEN 1 ELSE 0 END foobar, car FROM DUAL WHERE 1 = 1&quot;, reindent = True);'

<b>What is the expected output? What do you see instead?</b>
I would expect something like:

SELECT foo,
       bar,
       CASE
           WHEN foo < bar THEN 1
           ELSE 0
       END foobar,
       car
FROM DUAL
WHERE 1 = 1

but instead get:

SELECT foo, bar, CASE
                     WHEN foo < bar THEN 1
                     ELSE 0
                 END foobar,
                     car
FROM DUAL
WHERE 1 = 1

This isn't a problem for short lists of select clauses like this example, but makes the output nearly unreadable for long lists of select clauses.

<b>What version of the product are you using? On what operating system?</b>
Tested with both 0.1.3 and head on CentOS 5.4, python 2.7.

<b>What steps will reproduce the problem?</b>
1. Upload the attached file at sqlformat.appspot.com
2. Click &quot;Format&quot;
3. 500

<b>What is the expected output? What do you see instead?</b>


<b>Please use labels and text to provide additional information.</b>

<b>What steps will reproduce the problem?</b>
Run the command:
python -c 'import sqlparse; print sqlparse.format(&quot;SELECT foo.foo, foo.key, foo.bar FROM foo WHERE 1 = 1&quot;, reindent = True);'

<b>What is the expected output? What do you see instead?</b>
I would expect something like:

SELECT foo.foo,
       foo.key,
       foo.bar
FROM foo
WHERE 1 = 1

but instead get:

SELECT foo.foo,
       foo.key, foo.bar
FROM foo
WHERE 1 = 1

I suspect that the parser is interpreting the &quot;key&quot; of foo.key as a keyword and is getting confused as a result.

<b>What version of the product are you using? On what operating system?</b>
Tested with both 0.1.3 and head on CentOS 5.4, python 2.7.

sqlparse thinks that "?" is an operator, and treats ":a" as a piece of punctuation and then an identifier.  "$1" is an error.

It would be nice if sqlparse supported at least one form of placeholder syntax.

Django is too slow for the first request and maybe a bit too much framework 
for this little app too :)

<b>What steps will reproduce the problem?</b>

>>> import sqlparse
>>> p = sqlparse.parse('select foo')[0]
[<DML 'select' at 0xb70d0e0c>,
 <Whitespace ' ' at 0xb70d0e4c>,
 <Identifier 'foo' at 0xb70d241c>]
>>> p.tokens[-1].value is None
<<< True 
>>> str(p.tokens[-1])
<<< 'foo'

<b>What is the expected output? What do you see instead?</b>
Identifier.value should be 'foo' too.

<b>Please use labels and text to provide additional information.</b>

"create table a (b integer not null, c integer)" is parsed incorrectly; sqlparse sees an IdentifierList of 'null, c'.

"create table a (b integer, c integer not null)", however, gets parsed correctly, because there's no comma after the NULL.

This also happens with a 'references x' constraint on the first column, except then the IdentifierList contains the name of the column and the next column.

Consider the results when sqlparse parses the string:

    "select (3 + 4) = 7 and (5 + 6) = 11;"

It identifies "3 + 4 = 7" as a "Comparison", which is questionable; it's really an expression.

More importantly, sqlparse doesn't identify the outer 'and' expression as a Comparison of its own, either; it just sees Comparison, Keyword, Comparison.

Finally I'd like to be able to strip out the parenthesis tokens and easily see just the expression structure, rather than the syntactic structure.  (Don't get me wrong, being able to preserve byte-for-byte input is great; it's just not always what is most of interest when I'm processing stuff.)

Format this with sqlformat -r -ilower -:
select 
    timestamp::date,
    count(*)
from logs
where 
    db_id = 123
    and user_id = 123
    and timestamp between '2010-05-01'::timestamp and '2010-05-30'::timestamp
group by timestamp::date

Result would be:
SELECT TIMESTAMP::date, COUNT(*)
FROM logs
WHERE db_id = 123
  AND user_id = 123
  AND TIMESTAMP BETWEEN '2010-05-01'::TIMESTAMP
  AND '2010-05-30'::TIMESTAMP
GROUP BY TIMESTAMP::date

formatting clause and z between x and y
as 
'''
and z between x
and y
'''
obscures it instead of making it more readable

Currently the lexer unifies line breaks to UNIX LE. It should preserve the
line endings given in the original string (maybe even if it contains mixed
line endings).

Example statement:

select * from (select x.foo, x.bar as foo2, x.baz from mytable)

Results in:

SELECT *
FROM
  (SELECT x.foo,
          x.bar AS foo2,
                   x.baz
   FROM mytable)

This doesn't happen when the inner statement is formatted on it's own.

python -c "import sqlparse; print sqlparse.format('SELECT id, name FROM (SELECT id, name FROM bar)', reindent=True)"

SELECT id,
       name
FROM
  (SELECT id,
          name
   FROM bar)

python -c "import sqlparse; print sqlparse.format('SELECT id, name FROM (SELECT id, name FROM bar) as foo', reindent=True)"

SELECT id,
       name
FROM
  (SELECT id, name
   FROM bar) as foo


When aliasing the subselect the query is not formatted properly (no new line after id, ).

I tried to split with sqlparse.split() the following query:
'CREATE TEMPORARY TABLE tmp SELECT IF(a=1, a, b) AS o FROM one; SELECT t FROM two'

It returns only 1 query instead of 2:
[u'CREATE TEMPORARY TABLE tmp SELECT IF(a=1, a, b) AS o FROM one; SELECT t FROM two']

It is apparently a problem caused by the IF (without it, sqlparse returns 2 queries).

I expect:

>>> code = 'SELECT MAX(COLA) "a", COLB "b", COLC "c" FROM t;'
>>> print sqlparse.format(code, reindent=True)
SELECT MAX(COLA) "a",
       COLB "b",
       COLC "c"
FROM t;

I get:

>>> code = 'SELECT MAX(COLA) "a", COLB "b", COLC "c" FROM t;'
>>> print sqlparse.format(code, reindent=True)
SELECT MAX(COLA) "a",
                 COLB "b",
                 COLC "c"
FROM t;

It gets even worse when more than one columns are wrapped in a function call.

>>> code = 'SELECT MAX(COLA) "a", MAX(COLB) "b", COLC "c" FROM t;'
>>> print sqlparse.format(code, reindent=True)
SELECT MAX(COLA) "a", MAX(COLB) "b",
                                COLC "c"
FROM t;


Software environment:
sqlparse 0.1.2
Python 2.6.6 (r266:84292, Jan  4 2011, 16:45:45) 
Mac OS X 10.6.7

<b>What steps will reproduce the problem?</b>
>>> import sqlparse
>>> sqlparse.format(&quot;select * from foo where bar = 1 limit 1&quot;, reindent=True)
<<< u'select *\nfrom foo\nwhere bar = 1limit 1'


<b>What is the expected output? What do you see instead?</b>

Output should read u'select *\nfrom foo\nwhere bar = 1 limit 1' (space before limit)


<b>Please use labels and text to provide additional information.</b>

Reindent filter has a lousy performance on certain statements. One of the "statements of death" is attached.

The problem is that ReindentFilter._get_offset() is a performance hog. The number of calls to flatten() in offset calculation is totally insane. 

Possible solutions are:
- remember offsets
- limit flatten() range
- reimplement reindent using a different approach

One problem is that the statement changes constantly in this phase of processing and I'm not sure if it makes sense to store some kind of state at all.

NB, this causes DeadlineExceeded errors on App Engine

<b>What steps will reproduce the problem?</b>
1. parse a CREATE statement with a field using LONG as type
<b>2.</b>
<b>3.</b>

<b>What is the expected output? What do you see instead?</b>
LONG recognized as Name.Builtin, instead it's not recognized

<b>What version of the product are you using? On what operating system?</b>
0.1.1

<b>Please provide any additional information below.</b>

<b>What steps will reproduce the problem?</b>
1. Parse a query with a nvl statement in a where or and. E.g.:

   SELECT *    
     FROM table1 tab1
     JOIN table2 tab2 ON tab1.id = tab2.id
    WHERE SYSDATE BETWEEN tab1.date1 AND NVL(tab2.date2,SYSDATE)

<b>What is the expected output? What do you see instead?</b>
Expected to find that &quot;tab1.id&quot; and &quot;tab2.id&quot; would be parsed as Identifier instead they are split into two tokens with type name and a token with type punctuation.

<b>What version of the product are you using? On what operating system?</b>
sqlparse 0.1.2 (from Python Package Index)
python 2.7.1
windows xp

<b>What steps will reproduce the problem?</b>
1. Run the attached file

<b>What is the expected output? What do you see instead?</b>
I expect to see the file formatting according to the default formatting options.  Instead I get a 500 error.


<b>What version of the product are you using? On what operating system?</b>
I was using the free online version on a Windows Server 2003 server.

<b>Please provide any additional information below.</b>

<b>What steps will reproduce the problem?</b>
1. run the following script

import sqlparse
t = sqlparse.parse(&quot;create&quot;)[0].token_first()
t.match(&quot;CREATE&quot;) #returns false, doesn't make sense


<b>What is the expected output? What do you see instead?</b>

Last line should return true, DDL is still a keyword


<b>What version of the product are you using? On what operating system?</b>

latest release, on linux


<b>Please provide any additional information below.</b>

The problem is, that match ignores case only for Keyword, it should check if Keyword is one of your descendants.

"create table a (ended timestamp)" or "create table a (created timestamp)" results in a keyword token 'end' and then an identifier token 'ed', or 'create', 'd' respectively.

This can be fixed by adjusting the 'END' and 'CREATE' regular expressions in Lexer.tokens['root'] to begin and end with '\b', to require word boundaries.

<b>What steps will reproduce the problem?</b>
1. Format a statement with reindent=True and indent_tabs=True

<b>What is the expected output? What do you see instead?</b>

Expected output is a nicely formatted statement, but the formatter
calulates indentations based on spaces. Maybe indentation using tabs will
be removed.


<b>Please use labels and text to provide additional information.</b>

Example statement:

select null, foo, bar from mytable;

"null" should be added to identifier list, so that e.g. formatting with
reindent returns

select null,
       foo,
       bar
from mytable;

This is a result of interactive session:

In [35]: sqlparse.parse('''CREATE TABLE auth_user_user_permissions ("id" integer NOT NULL PRIMARY KEY,
    "user_id" integer NOT NULL,
    "permission_id" integer NOT NULL REFERENCES "auth_permission" ("id"),
    UNIQUE ("user_id", "permission_id")
)
''')
Out[40]: (<Statement 'CREATE...' at 0xb7217d4c>,)

In [41]: x = _[0].tokens

In [42]: x
Out[42]: 
[<DDL 'CREATE' at 0xb722048c>,
 <Whitespace ' ' at 0xb72204ac>,
 <Keyword 'TABLE' at 0xb72204cc>,
 <Whitespace ' ' at 0xb72204ec>,
 <Function 'auth_u...' at 0xb721f1b4>,
 <Newline ' ' at 0xb7220bcc>]

I wanted to parse columns from this statement, but none was returned.

v0.1.1
Using reindent = True

At some stage I'll dig into the code to find the problem here, but if
someone fixes it before I get around to it that would be wonderful. :)

Formatted SQL is:

SELECT SERIAL# SERIAL,
       SERVICE_NAME SERVICE_NAME,
       P1 WAIT_P1,
       P2 WAIT_P2,
       P3 WAIT_P3,
       ROW_WAIT_ROW# ROW_WAIT_ROW,
       ROW_WAIT_OBJ# ROW_WAIT_OBJ,
       ROW_WAIT_BLOCK# ROW_WAIT_BLOCK,
       ROW_WAIT_FILE# ROW_WAIT_FILE,
       DECODE(STATE,'WAITING',
                    'WAITING',
                    'ON CPU') WAIT_STATE,
                              DECODE(STATE,'WAITING',
                                           WAIT_CLASS,
                                           'ON CPU') WAIT_CLASS,
                                                     DECODE(STATE,'WAITING',
                                                                  EVENT,
                                                                  'ON CPU')
WAIT_EVENT, SID SID, AUDSID AUDSID, USERNAME USERNAME, MACHINE MACHINE,
CLIENT_IDENTIFIER CLIENT_IDENTIFIER, COMMAND COMMAND, STATUS STATUS,
PROCESS PROCESS, PROGRAM PROGRAM, TYPE TYPE, CLIENT_INFO CLIENT_INFO,



                                     SQL_HASH_VALUE SQL_HASH_VALUE,



                                     SQL_ID SQL_ID,



                                     PLAN_HASH_VALUE PLAN_HASH_VALUE,



                                     MODULE MODULE,



                                     ACTION ACTION,



                                     LOGON_TIME LOGON_TIME,



                                     LAST_CALL_ET LAST_CALL_ET,



                                     FIXED_TABLE_SEQUENCE FIXED_TABLE_SEQUENCE,



                                     WAIT_CLASS WAIT_CLASS,



                                     SECONDS_IN_WAIT SECONDS_IN_WAIT,



                                     WAIT_TIME WAIT_TIME,



                                     PLSQL_ENTRY_OBJECT_ID
PLSQL_ENTRY_OBJECT_ID,



                                     PLSQL_ENTRY_SUBPROGRAM_ID
PLSQL_ENTRY_SUBPROGRAM_ID,



                                     PLSQL_OBJECT_ID PLSQ

<b>What steps will reproduce the problem?</b>
1. end a field definition with CONSTRAINT constraint_name NOT NULL,
2. start the next field with field_name
<b>3.</b>

<b>What is the expected output? What do you see instead?</b>
&quot;NULL&quot; &quot;,&quot; and &quot;field_name&quot; as separate tokens, however you get an IdentifierList instead

<b>What version of the product are you using? On what operating system?</b>
0.1.1

<b>Please provide any additional information below.</b>

Hi,

python -c "import sqlparse; print sqlparse.format('select foo.id, foo.name from user', reindent=True)"
select foo.id,
       foo.name
from user


python -c "import sqlparse; print sqlparse.format('select user.id, user.name from user', reindent=True)"
select user.id, user.name
from user


The second query is not formatted properly (no newline after user.id) because sqlparse thinks user is a keyword which it is not the case for mysql.

One way to fix this would be to quote the table name:

python -c 'import sqlparse; print sqlparse.format("""select "user".id, "user".name from user""", reindent=True)'
select "user".id,
       "user".name
from user


But I can't do that because my queries are automatically generated by SQLAlchemy, instead I monkey patched sqlparse to use the list of keywords provided by SQLAlchemy in sqlalchemy.dialects.mysql.base

Would you consider a patch adding a dialect option to sqlparse ?

>>> import sqlparse
u'SELECT * FROM user; -- why!\n-- hello\n'
>>> sqlparse.format('-- hello\nSELECT * FROM user;\n-- hello\n', strip_comments=True)
u'SELECT * FROM user;\n-- hello\n'
>>> sqlparse.format('-- hello\nSELECT * FROM user; -- hello\n-- hello\n', strip_comments=True)
u'SELECT * FROM user; -- hello\n-- hello\n'

Making the following change fixes the issue

diff -r 96cf90d0f248 sqlparse/engine/grouping.py
--- a/sqlparse/engine/grouping.py   Wed Apr 13 16:04:56 2011 +0200
+++ b/sqlparse/engine/grouping.py   Fri Jun 10 10:54:34 2011 +0100
@@ -229,6 +229,7 @@
                                        [lambda t: t.ttype in T.Comment,
                                         lambda t: t.is_whitespace()])
         if end is None:
+            tlist.group_tokens(sql.Comment, [token])
             idx = tidx + 1
         else:
             eidx = tlist.token_index(end)

<b>What steps will reproduce the problem?</b>
python -c 'import oldsqlparse; print oldsqlparse.format(&quot;SELECT foo FROM bar,role WHERE 1 = 1&quot;, reindent = True);'

<b>What is the expected output? What do you see instead?</b>
I would expect something like:

SELECT foo
FROM bar,
     role
WHERE 1 = 1

but get:

SELECT foo
FROM bar,role
WHERE 1 = 1

instead.

<b>What version of the product are you using? On what operating system?</b>
Tested with both 0.1.3 and head on CentOS 5.4, python 2.7.

<b>Please provide any additional information below.</b>
I suspect &quot;role&quot; is being interpreted as some sort of keyword here.

extras/py3k/fixes.diff contains two fixes that need to be applied when using 
the 2to3 tool to produce a Python 3 version of sqlparse. These fixes should 
go into sqlparse itself to avoid extra patching.

<b>What steps will reproduce the problem?</b>
Take the following Statements:
insert into database_address (id, street, zip, city, state, country_id, 
comment ) values(41, '7, Rue De L\'Eperon', '75006','Paris','', (select id 
from database_country where name = 'France'), 'Box/Building: ');
insert into database_address (id, street, zip, city, state, country_id, 
comment ) values(42, 'Eisenbahnstr. 41', '79098','Freiburg','', (select id 
from database_country where name = 'Germany'), 'Box/Building: ');
insert into database_address (id, street, zip, city, state, country_id, 
comment ) values(43, 'Grillparzerstr. 46', '81675','München','', (select id 
from database_country where name = 'Germany'), 'Box/Building: ');

<b>What is the expected output? What do you see instead?</b>
I would expect thst these statements are parsed as 3 seperate statements.

<b>What version of the product are you using? On what operating system?</b>
Version: Not sure. 
OS: Ubuntu 9.04

<b>What steps will reproduce the problem?</b>

sqlparse.format(&quot;create table created_test ( name text, created date )&quot;, 
    keyword_case='upper', 
    identifier_case='lower')

<b>What is the expected output? What do you see instead?</b>

expected:

u'CREATE TABLE created_test ( name text, created date )'

actual:

u'CREATE TABLE CREATEd_test ( name text, CREATEd date )'


<b>What version of the product are you using? On what operating system?</b>
Mac OS X 10.5/10.6

$ sudo easy_install sqlparse
Password:
Searching for sqlparse
Best match: sqlparse 0.1.1
Processing sqlparse-0.1.1-py2.6.egg
sqlparse 0.1.1 is already the active version in easy-install.pth
Installing sqlformat script to /usr/local/bin

Using /Library/Python/2.6/site-packages/sqlparse-0.1.1-py2.6.egg
Processing dependencies for sqlparse
Finished processing dependencies for sqlparse

<b>Please provide any additional information below.</b>

I was able to get the expected results, without seeming to break anything else, with the following:

--- sqlparse-0.1.1/sqlparse/lexer.py    2009-05-05 21:59:02.000000000 -0700
+++ sqlparse-0.1.1-mod/sqlparse/lexer.py    2009-06-23 16:27:45.000000000 -0700
@@ -174,7 +174,7 @@
             (r'&quot;(&quot;&quot;|[^&quot;])*&quot;', String.Symbol), # not a real string literal in ANSI SQL
             (r'(LEFT |RIGHT )?(INNER |OUTER )?JOIN', Keyword),
             (r'END( IF| LOOP)?', Keyword),
-            (r'CREATE( OR REPLACE)?', Keyword.DDL),
+            (r'CREATE( OR REPLACE)?\b', Keyword.DDL),
             (r'[a-zA-Z_][a-zA-Z0-9_]*', is_keyword),
             (r'\$([a-zA-Z_][a-zA-Z0-9_]*)?\$', Name.Builtin),
             (r'[;:()\[\],\.]', Punctuation),

Please note: I'm very nearly completely parser-illiterate, and so can't even begin to guess at whether that was even remotely 
the right thing to do, but it *seemed* to work for me