itsjohncs / phial Goto Github PK

I should get tox to build a few test sites. I just changed the default value of the --source option for #14 and I didn't break any tests, which seems like a bad thing.

The loggers module isn't structured very nicely and could use a good touch-up.

The tracebacks that Phial spills out when there's an exception in the user's code does not include any of their code.

Traceback (most recent call last):
  File "/home/john/Projects/phial/phial/tool.py", line 333, in build_app
    output_dir = output_dir, index_path = index_path)
  File "/home/john/Projects/phial/phial/processor.py", line 55, in process
    artifacts = render_site(source_dir)
  File "/home/john/Projects/phial/phial/processor.py", line 165, in render_site
    artifacts += render_page(i)
  File "/home/john/Projects/phial/phial/processor.py", line 370, in render_page
    result = page.func(**filter_args(potential_args))
  File "app.py", line 41, in post_page
TypeError: 'NoneType' object does not support item assignment

Notice the lack of code under the last frame.

As the command queue gets more awesome it's tempting to turn Phial into a one-stop shop for all your build needs, and transform it into something like Gulp.js (which it shares a lot in common with). I think it would be very hard to market Phial as both an awesome microframework for building static sites, and an awesome general-purpose build system. There's also syntactical changes and whole areas of the API that would be unneeded depending on how your developing.

For this reason, I think it might be a good idea to plan on creating a separate project, Gulpy, that actually tries to be a Python replacement for Gulp.js. It could just include phial as a submodule inside of itself, and wrap it and expand on it as needed.

A lot of the tools users will want to use in their pipelines come as executables and not necessarily as nice Python libraries. It would be beneficial for all of them to have a consistent interface. The best way to do that, I think, is to create a useful base class and promote it.

I imagine something like this:

@pipeline("js/*")
def js(source):
    return source | move("res/js/")

What if, instead of having the processor module go through things after all the page functions have been registered, they're executed as they're seen? Would the code become simpler?

If the user passes -o :temp: I'd like a temporary directory to be created that is output to and then destroyed when the tool exits. This will be useful when combined with the server and monitor options.

The options related to the monitoring mode and the serving mode can be grouped. Later on I'll probably have grouping for the deploy-to-git feature and any other single feature with multiple arguments involved.

I have high hopes for this project. I think it's pretty cool and I've gotten excited reactions from people when I've told them about the idea. My hope is that it can be a nice project to contribute to as well as use though. Towards that, I should make it as straightforward as possible for somebody to jump in and contribute a patch/open an issue.

A Contributing to Phial document would go a long way I think. I'll also want to define the sorts of things I care about. For example, I think unit testing for the user-facing API should be mandatory, whereas anything internal doesn't necessarily need to be unit tested.

Describing how to set up the environment for development (and making that as easy as possible) will also be a good idea. This'll be a long process and I expect this thread to continue well into the foreseeable future.

I don't think my motivations were in the right place when I created the idea of the source directory. My thoughts were that you could make an app.py file and swap out the source files easily. And while this is true, you could do this anyways without phial mucking up your day.

The idea here is to allow you to feed the output of your app into another set of pipelines and pages. The default phase would be 0, and you could add a phase decorator to a pipeline or page to make it run either before or after some other stuff (so -1 would be a pre-phase and 1 would be a post).

It's pretty annoying right now to deal with the different-logger dependency. I'm fairly sure I broke things with some recent changes to different-logger too, so I'll need to do point at an older sha until things are fixed.

It has been difficult deciding what to do and what not to do at times. I'm hoping some core principles will go a long way towards improving this situation. Here's some potentials:

• Upgrading Phial (within a major version) will never break your site.
• Making a non-English site is as easy as making an English one.
• Phial's feature-set is the lowest common denominator of the major SSGs.
• Develop for maximum flexibility first, then add shortcuts for the most common cases.
• Phial is for Python developers.

site/ conflicts with Jekyll's _site/ directory which is actually its output directory. I don't think this is going to be friendly to people. source/, src/, or data/ are all better options I think.

The ETE test sites are seriously buried: phial/phial/tests/test_resources/sites_dir/cats/. I should bring them out into a more visible location because they're fairly useful examples of the current state of Phial (I'm also tired of cd-ing down into that directory whenever I want to poke at it).

It would run the lint checks, but also give instructions on installing them (or just install them automatically, but that seems a little aggressive). I could also put it in a make file.

Currently you're forced to use globals to pass information between page functions. We could instead do something like...

@page(...)
def other_page():
    main_page.pass_data("my_key", "some data")

@page(..., passed_data=True)
def main_page(my_data):
    print my_data["my_key"]

I'm not sure if this is really any better than globals though. It could make things easier to unit test, but while I don't want to discourage unit testing, I really doubt anyones going to be unit testing their static site's page functions...

I think the benefit of using a requierments file is it allows people who clone your repo to use the project directly, without having to do the pip install/uninstall trick. I'll have to make the setup.py file grab and include the dependencies.

This will allow support for GitHub pages which is (I imagine) how most people will deploy their site. I'm deploying it that way anyways and I want this feature, so that's good enough for me 😉.

The processor and the way Phial actually does the builds is pretty unmaintainable. I'd like to scrap it and write it anew.

The current one is legitimately broken, so of course anything would be an improvement. But I'd like something that's as nice as possible and seems to fit in well with the rest of Phial's interface.

I think deploying to the cheese shop is sufficient for now. I may want to get fancier using Superzippy and/or some other freezing solution in the future though.

I have it set up for johncs.com and it's pretty great.

This was originally thought of as a feature to be tied into #4 but I think it will be useful outside of that as well. This would allow intelligent updating of an output directory.

Unless disabled, a .phial_index file will be generated in the output directory that contains a listing of every file that Phial generated during the last build. Then whenever Phial goes to build into an output directory it checks that file and deletes any files and folders that were generated last time.

I'm not sure if this should go into this repo or a new one though...

• Detected change in watch list. is a confusing log message.
• The default verbosity shouldn't include info messages, and the -v option should be accepted multiple times.

I'd like to use some introspection to pass the target into each page function if they have a target parameter. This won't be the first extra bit of information I'm sure, so I'd like to do this generally.

I'm open to another method of doing this as well. I just want to make sure that the next time I want to add more information it's not going to break all the sites that exist already.

I like the @page decorator but the handling of static files seems clunky. It's also super limited. If you want to do stuff like run you CSS files through SASS, Phial isn't going to make your life easy.

To keep things general I think making some kind of general pipeline API would be good. Following a pattern like Gulp.js's would be good since it's implementation seems to be very popular.

Something like this seems good:

@pipeline("res/css/*")
def less_pipeline(sources):
    return sources | less() | concat("all.css") | minify()

There's a project called webassets that looks useful. The API really doesn't appeal to me though, and doesn't seem to fit well with Phial. It does however have support for a whole lot of tools. It might be worthwhile to write a wrapper for it as a separate project.

The only thing in the Phial code base should be scaffolding for pipelines (and maybe some super basic pipelines like concat). The implementations should be in separate projects.

It would be even better if the pipelines are general enough to not necessarily need wrappers around stuff... I could imagine doing this with an object like this being passed in for sources:

class Pipeline(object):
    def __bitwise_or__(self, other): # almost certainly not the right name
        if not isinstance(other, Pipeline):
            for i in self.sources:
                output.append(other(i)) or something...

Remember to keep things as simple as possible though. We're bordering on making a magical feature, so we need to make sure it's all as straightforward as possible.

Looking at this again for the first time in awhile is giving me some good naive reader feedback. Namely, I notice that it's strange that @phial.pipeline takes in the source files as the first argument (though it doesn't have a second argument), while @phial.page takes in the output file as the first argument.

The idea that the output name has to be provided to Phial when the module is imported seems like a little much, but makes sense if I want to actually make this capable of being a build system. If that's the case, I should make people specify the output of a pipeline upfront. If I don't actually care about getting the output file name(s), I should drop them everywhere.

Since dropping them everywhere will probably shock people who are used to seeing build systems ask for this information (thank you make), I'm thinking the former is the best idea. Not sure though.

Phial, the library, is for developers. Everyone else can go away. Developers often want to make things for non-developers though, and it would be good if Phial could offer some support for that use case. I think this is really the main selling point of Wordpress, and is why it seems like only devs use Jekyll.

An officially supported, but separate tool is probably the way to go with this I think. It can be a GUI application for Windows and Mac (if they're using Linux and aren't comfortable running a simple shell script then too bad for them) and try to solve typical problems of end users. Like showing previews, showing where to put files, letting you edit the site in a way that makes sense to end users...

Anyways, just ideas at this point. Not even sure if it's worth doing this (but my gut tells me that this would be a huge selling point if I can do it well).

This is awesome:

@page("index.htm")
def home():
    return "Hello World!"

What's not awesome though, is the syntax for making multiple pages:

@page("posts/{name}.htm", files="posts/*.rst")
def post_page(source):
     # ...

I don't have any idea how {name} is filled in. I can kind of guess at what the files bit is doing, but I wouldn't think source was a Document object without reading the docs. I think I can find a better syntax.

@page("posts/{}.htm", foreach=["post1", "post2", "post3"])
def post_page(path, item):
    # ...

If I wanted to do the file enumeration I had before:

@page("posts/{}.htm", foreach=glob("posts/*.rst")):
def post_page(path, item):
    # ...

It's more verbose, but I think it's clearer and more powerful.

This will allow people to make single file Phial sites. It's also perfectly reasonable that someone would want their source files to be contained in the same directory as their app script.

I really like the terms of the Apache license, especially its comments on patents, but I think it's probably unnecessary. Nowadays anyone can be targeted by patent trolls regardless of what they're doing, so it may not be necessary to protect against the case of someone purposely putting something they have a patent for inside the source tree.

A BSD license or the MIT license may be more appropriate.

Below shows a run through during testing.

~/Projects/blog - [env] master(+0/-0,1)*
: 1 ± python -m phial --testing app.py
INFO - Created temporary directory at '/tmp/tmpqpJVB6'.
INFO - Building application from sources in './site' to output directory '/tmp/tmpqpJVB6'.
INFO - Starting server at http://localhost:9000
INFO - Entering monitor mode. Watch list: ['.', './site']. Don't watch list: ['/tmp/tmpqpJVB6'].
^C----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 35567)
INFO - User sent interrupt, exiting.
Traceback (most recent call last):
  File "/usr/lib64/python2.7/SocketServer.py", line 284, in _handle_request_noblock
INFO - Removing files/directories ['/tmp/tmpqpJVB6'].

I think the best way to handle this will involve using a pipe to trap the error output from the server and printing it out as a debug message.

(1) Phial will only deal with YAML and (2) it will also get upset if YAML isn't installed. I'd like to change the first part, but I think I really must change the second.

I don't think it makes much sense to create a plugin-like system to do this, but it could make sense to split things up a bit more so making your own parse_json_frontmatter function is easy. I could also make the current frontmatter parsing function take in a function or class that it would use to consume the frontmatter, but that seems kinda pointless...

Will think more on this.

Been doing lots of rapid breaking changes but I think things are settling down. It's about time to fix the tests.

The regression tests are failing because the index files aren't matching up. I don't think this is really something we want to include in the regression test though because the index file format isn't part of our API. I'd just disable the functionality and I can make a separate test for it later.

Might as well keep things consistent with Khan Academy so our preferred styles don't diverge... I'll just use the same tools.

In Documents.py we modify PyYAML's global state with the following code.

yaml.SafeLoader.add_constructor(
    u"tag:yaml.org,2002:str",
    lambda loader, node: loader.construct_scalar(node)
)

This will ensure that all of the strings returned by YAML's SafeLoader class are unicode objects (by default, it will give you a str if the string contains only ASCII characters). This will affect anyone using Phial which is definitely not something we should be doing. I must find an alternative method here.

Logging right now is pretty good, but there's a lot of noise. I imagine there's a lot that can be done.

After the user install phial they should be able to execute the phial command. Currently they can't and they have to use python -m phial.

Currently any and all configuration options must be passed through the command line. The output directory and source direct for example. While this is fine and should be allowed, you should also be able to set these things in your code.

Currently Phial just recurses into the directory and makes a hash of all the names and metadata. I have a feeling this won't scale too well for large sets of files (but maybe it will?).

Gulp.js does this, and though I think I should avoid feeling too similar to it (since its solving a different problem), it might not be a bad idea to have task names. I can grab them from the functions that are registered with Phial.

Coming back to this after forgetting the syntax has once again proved valuable. The syntax for the @page decorator is not intuitive to me. It's very unclear that the extension and directory information is being cut off by some magical filter. Let's use the cats app as an example:

@phial.page("cats/{0}.htm", foreach="cats/*")
def bio_page(target, item):
    frontmatter, content = phial.parse_frontmatter(item)

    cats.append({"target": target, "name": frontmatter["name"]})

    template = phial.open_file("bio_template.htm").read()
    return template.format(content=content.read(), **frontmatter)

It's not super intuitive what's going on here. It's fine enough after you figure it out (which might be good enough), but this library would be a lot more attractive if users thought "of course!" immediately after looking at the syntax.